3560 boot failure

[rcoote5902_2]

I have a 3560-24PS switch that began constantly rebooting this morning and I'm unable to get it to stop.  Here's an output of what it's doing:

Is this fixable?

Any assistance is appreciated.

Rob

Initializing flashfs...

flashfs[1]: 193 files, 3 directories
flashfs[1]: 0 orphaned files, 0 orphaned directories
flashfs[1]: Total bytes: 15998976
flashfs[1]: Bytes used: 15998976
flashfs[1]: Bytes available: 0
flashfs[1]: flashfs fsck took 2 seconds.
flashfs[1]: Initialization complete....done Initializing flashfs.
Checking for Bootloader upgrade.. not needed

POST: CPU MIC register Tests : Begin
POST: CPU MIC register Tests : End, Status Passed

POST: PortASIC Memory Tests : Begin
POST: PortASIC Memory Tests : End, Status Passed

POST: CPU MIC interface Loopback Tests : Begin
POST: CPU MIC interface Loopback Tests : End, Status Passed

POST: PortASIC RingLoopback Tests : Begin
POST: PortASIC RingLoopback Tests : End, Status Passed

POST: Inline Power Controller Tests : Begin
POST: Inline Power Controller Tests : End, Status Passed

POST: PortASIC CAM Subsystem Tests : Begin
TCAM (asic:0) walking-one test failed
  Write: 80_00000000_00000000-00_00000000_00020000  
  Read:  80_20200000_00000000-00_00000000_00020000  
POST: POST Failed
POST: PortASIC CAM Subsystem Tests : End, Status Failed

POST: CAM test failed
POST Failed: disabling stack links and shutting down SDP

driver class subsystem initialization failed

*Mar  1 00:00:15.879: %SYS-3-LOGGER_FLUSHING: System pausing to ensure console debugging output.

*Mar  1 00:00:15.879: %SYS-2-ASSERTION_FAILED: Assertion failed: "reg_used_xdr_portid_to_lc_logical_slot()" -Process= "Init", ipl= 0, pid= 3
-Traceback= CB7318 CB7A64 C24938 1E903C 1E922C 14D6E94 14D6F2C CC2FD8 CC3214 A299C8 A20498

%Software-forced reload

00:00:16 UTC Mon Mar 1 1993: Unexpected exception to CPUvector 2000, PC = A27A3C 
-Traceback= A27A3C A27A00 1E903C 1E922C 14D6E94 14D6F2C CC2FD8 CC3214 A299C8 A20498

=== Flushing messages (00:00:16 UTC Mon Mar 1 1993) ===

Buffered messages:
Queued messages:
*Mar  1 00:00:16.676: %SYS-3-LOGGER_FLUSHING: System pausing to ensure console debugging output.

*Mar  1 00:00:16.299: %SYS-3-LOGGER_FLUSHED: System was paused for 00:00:00 to ensure console debugging output.

*Mar  1 00:00:16.332: %SYS-3-MGDTIMER: Uninitialized timer, init with uninitialized master, timer = 2681950. -Process= "Init", ipl= 0, pid= 3
-Traceback= CB7318 CB7A64 14CEB90 14CEC64 14CEDE0 11093C8 11032F8 1105910 1105A80 1262AC8 1263C08 12611A4 1261258 1268DF0 A27520 A27A00
*Mar  1 00:00:16.332: NRGYZ:ERROR:Socket creation failed for UDP
*Mar  1 00:00:16.332: NRGYZ:ERROR:Socket connect fail for 255.255.255.255:0(13)
*Mar  1 00:00:16.550: %Insufficient flash filespace for extended crashinfo.
Cisco IOS Software, C3560 Software (C3560-IPBASEK9-M), Version 12.2(50)SE1, RELEASE SOFTWARE (fc2)
Copyright (c) 1986-2009 by Cisco Systems, Inc.
Compiled Mon 06-Apr-09 08:00 by amvarma

Debug Exception (Could be NULL pointer dereference) Exception (0x2000)!
SRR0 = 0x00695AAC  SRR1 = 0x00029230  SRR2 = 0x00694B94  SRR3 = 0x00021000
ESR = 0x00000000  DEAR = 0x00000000  TSR = 0x8C000000  DBSR = 0x10000000

CPU Register Context:
Vector = 0x00002000  PC = 0x00A27A3C  MSR = 0x00029230  CR = 0x39000053
LR = 0x00A27A00  CTR = 0x014F4BB8  XER = 0xE0000000
R0 = 0x00A27A00  R1 = 0x026EB760  R2 = 0x00000000  R3 = 0x037F4564
R4 = 0x0000005A  R5 = 0x00000005  R6 = 0x00000000  R7 = 0x0000005A
R8 = 0x002D8700  R9 = 0x00000000  R10 = 0x0000005A  R11 = 0x023C7C40
R12 = 0xA2F88180  R13 = 0x00110000  R14 = 0x00CC31B4  R15 = 0x00000000
R16 = 0x00000000  R17 = 0x00000000  R18 = 0x00000000  R19 = 0x00000000
R20 = 0x00000000  R21 = 0x00000000  R22 = 0x00000000  R23 = 0x00000000
R24 = 0x00000000  R25 = 0x00000000  R26 = 0x00000000  R27 = 0x00000001
R28 = 0x000004A5  R29 = 0x0191B180  R30 = 0x01AA94A8  R31 = 0x00000000

Stack trace:
PC = 0x00A27A3C, SP = 0x026EB760
Frame 00: SP = 0x026EB770    PC = 0x00A27A00
Frame 01: SP = 0x026EBA38    PC = 0x00C24A14
Frame 02: SP = 0x026EBA50    PC = 0x001E903C
Frame 03: SP = 0x026EBAA0    PC = 0x001E922C
Frame 04: SP = 0x026EBAA8    PC = 0x014D6E94
Frame 05: SP = 0x026EBAC0    PC = 0x014D6F2C
Frame 06: SP = 0x026EBAE0    PC = 0x00CC2FD8
Frame 07: SP = 0x026EBBE8    PC = 0x00CC3214
Frame 08: SP = 0x026EBBF0    PC = 0x00A299C8
Frame 09: SP = 0x00000000    PC = 0x00A20498

Switch uptime is 19 seconds

cisco WS-C3560-24PS (PowerPC405) processor (revision G0) with 122880K/8184K bytes of memory.
Processor board ID CAT0836X1RX

[Leo Laohoo]

The switch is in a state called a boot-n-crash loop.  In this state, the switch boots up, encounters a problem, creates a crashlog info file/subdirectory, reboots and repeats the cycle.  Now somewhere during the process, the issue was "rectified" but  the flash memory is full and the switch couldn't write any updates of the crashlog so it crashes once more.

What you need to do is to power off the switch for about 10-15 seconds.  Power it up.  If the power comes back up, you need to delete the crashinfo subdirectory (command:  delete /f  /r flash:).

Hope this helps.  Please don't forget to rate our posts.

[rahurao]

Hi Rob,

I tried to check the issue and i can see that the switch is reloading while it is running. i decoded the tracebacks for you and found 4 bugs whic are hitting the image which you are using. I cannot share the bug id's with you as they are internal and confidential to Cisco.

0xCB7318:verrmsg(0xcb71c0)+0x158
0xCB7A64:errmsg(0xcb79b4)+0xb0
0x14CEB90:mgd_timer_complain_uninit(0x14ceb5c)+0x34
0x14CEC64:mgd_timer_init_common(0x14cec28)+0x3c
0x14CEDE0:mgd_timer_init_leaf(0x14cedac)+0x34
0x11093C8:sock_timer_init_leaf(0x1109340)+0x88
0x11032F8:add_sock_for_proc(0x1103160)+0x198
0x1105910:socket_create(0x1105828)+0xe8
0x1105A80:socket_open(0x11059c4)+0xbc
0x1262AC8:pn_udp_tx(0x1262a80)+0x48
0x1263C08:cpqr_xport_tx(0x1263b78)+0x90
0x12611A4:pn_udp_discovery_init(0x12610f8)+0xac
0x1261258:pn_discovery_init(0x1261218)+0x40
0x1268DF0:powernet_crash_hook(0x1268dd8)+0x18
0xA27520:run_crash_hook(0xa274ac)+0x74
0xA27A00:crashdump(0xa27980)+0x80

That tells us to upgrade the switch to the latest version to fix the issue.