Issue: the switch is rebooting continuously

frbachel05 · ‎10-08-2013

Hi everybody,

I have a trouble with a switch (switch 1 master of 3 switchs in one stack) that is rebooting continuously when i connect a machine on it.

Historic:

I rebooted the stack because of electric cut.

After startup completed, the switch 1 master rebooted again and again

I turned off the switch 1 and I downgraded the priority level

I powered on the swtich and it has been added to the stack with slave position.

If i don't connect PC on it, it's ok but as soon as i connect a PC, the switch is rebooting

When i connect or remove a PC on the switch, it reboot with logs below:

*Mar 1 00:00:34.342: %STACKMGR-4-SWITCH_ADDED: Switch 1 has been ADDED to the stack
*Mar 1 00:00:46.967: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan1, changed state to down
*Mar 1 00:00:48.309: %SPANTREE-5-EXTENDED_SYSID: Extended SysId enabled for type vlan
*Mar 1 00:00:51: %SYS-6-CLOCKUPDATE: System clock has been updated from 00:00:51 UTC Mon Mar 1 1993 to 01:00:51 MET Mon Mar 1 1993, configured from console by console.
*Mar 1 00:00:51: %SYS-6-CLOCKUPDATE: System clock has been updated from 01:00:51 MET Mon Mar 1 1993 to 01:00:51 MET Mon Mar 1 1993, configured from console by console.
*Mar 1 00:01:22: %SYS-5-CONFIG_I: Configured from memory by console
*Mar 1 00:01:23: %STACKMGR-5-SWITCH_READY: Switch 1 is READY
*Mar 1 00:01:23: %STACKMGR-4-STACK_LINK_CHANGE: Stack Port 1 Switch 1 has changed to state DOWN
*Mar 1 00:01:23: %STACKMGR-4-STACK_LINK_CHANGE: Stack Port 2 Switch 1 has changed to state DOWN
*Mar 1 00:01:24: %SYS-5-RESTART: System restarted --
Cisco IOS Software, C3750 Software (C3750-IPBASEK9-M), Version 12.2(52)SE, RELEASE SOFTWARE (fc3)
Copyright (c) 1986-2009 by Cisco Systems, Inc.
Compiled Fri 25-Sep-09 08:32 by sasyamal
*Mar 1 00:01:24: %STACKMGR-5-MASTER_READY: Master Switch 1 is READY
*Mar 1 00:01:24: %SSH-5-ENABLED: SSH 1.99 has been enabled
*Mar 1 00:01:25: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 10.68.248.7 Port 514 started - CLI initiated
*Mar 1 00:01:28: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 10.68.160.50 Port 514 started - CLI initiated
*Mar 1 00:01:53: %PARSER-6-VIEW_SWITCH: successfully set to view 'root'.
FRNTE-E2-12-CS3750-01#
*Mar 1 00:02:26: %AUTHMGR-5-START: Starting 'mab' for client (f0de.f158.3b0c) on Interface Fa1/0/33 AuditSessionID 0A44E11E0000000000023D25
FRNTE-E2-12-CS3750-01#
*Mar 1 00:02:28: %LINK-3-UPDOWN: Interface FastEthernet1/0/33, changed state to up
*Mar 1 00:02:29: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet1/0/33, changed state to up
FRNTE-E2-12-CS3750-01#

01:02:40 MET Mon Mar 1 1993: Unexpected exception to CPUvector 2000, PC = 1F03708
-Traceback= 1F03708

=== Flushing messages (01:02:40 MET Mon Mar 1 1993) ===

Buffered messages:

*Mar 1 00:00:34.342: %STACKMGR-4-SWITCH_ADDED: Switch 1 has been ADDED to the stack
*Mar 1 00:00:46.967: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan1, changed state to down
*Mar 1 00:00:48.309: %SPANTREE-5-EXTENDED_SYSID: Extended SysId enabled for type vlan
*Mar 1 00:00:51: %SYS-6-CLOCKUPDATE: System clock has been updated from 00:00:51 UTC Mon Mar 1 1993 to 01:00:51 MET Mon Mar 1 1993, configured from console by console.
*Mar 1 00:00:51: %SYS-6-CLOCKUPDATE: System clock has been updated from 01:00:51 MET Mon Mar 1 1993 to 01:00:51 MET Mon Mar 1 1993, configured from console by console.
*Mar 1 00:01:22: %SYS-5-CONFIG_I: Configured from memory by console
*Mar 1 00:01:23: %STACKMGR-5-SWITCH_READY: Switch 1 is READY
*Mar 1 00:01:23: %STACKMGR-4-STACK_LINK_CHANGE: Stack Port 1 Switch 1 has changed to state DOWN
*Mar 1 00:01:23: %STACKMGR-4-STACK_LINK_CHANGE: Stack Port 2 Switch 1 has changed to state DOWN
*Mar 1 00:01:24: %SYS-5-RESTART: System restarted --
Cisco IOS Software, C3750 Software (C3750-IPBASEK9-M), Version 12.2(52)SE, RELEASE SOFTWARE (fc3)
Copyright (c) 1986-2009 by Cisco Systems, Inc.
Compiled Fri 25-Sep-09 08:32 by sasyamal
*Mar 1 00:01:24: %STACKMGR-5-MASTER_READY: Master Switch 1 is READY
*Mar 1 00:01:24: %SSH-5-ENABLED: SSH 1.99 has been enabled
*Mar 1 00:01:25: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 10.68.248.7 Port 514 started - CLI initiated
*Mar 1 00:01:28: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 10.68.160.50 Port 514 started - CLI initiated
*Mar 1 00:01:53: %PARSER-6-VIEW_SWITCH: successfully set to view 'root'.
*Mar 1 00:02:26: %AUTHMGR-5-START: Starting 'mab' for client (f0de.f158.3b0c) on Interface Fa1/0/33 AuditSessionID 0A44E11E0000000000023D25
*Mar 1 00:02:28: %LINK-3-UPDOWN: Interface FastEthernet1/0/33, changed state to up
*Mar 1 00:02:29: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet1/0/33, changed state to up
Queued messages:
*Mar 1 00:02:43: %SYS-3-LOGGER_FLUSHING: System pausing to ensure console debugging output.

*Mar 1 00:02:40: %Insufficient flash filespace for extended crashinfo.
Cisco IOS Software, C3750 Software (C3750-IPBASEK9-M), Version 12.2(52)SE, RELEASE SOFTWARE (fc3)
Copyright (c) 1986-2009 by Cisco Systems, Inc.
Compiled Fri 25-Sep-09 08:32 by sasyamal

Debug Exception (Could be NULL pointer dereference) Exception (0x2000)!
SRR0 = 0x018BA7A0 SRR1 = 0x00029230 SRR2 = 0x017613DC SRR3 = 0x00021000
ESR = 0x00000000 DEAR = 0x00000000 TSR = 0x8C000000 DBSR = 0x00800000

CPU Register Context:
Vector = 0x00002000 PC = 0x01F03708 MSR = 0x00029230 CR = 0x30000095
LR = 0x0191B180 CTR = 0x0178AD2C XER = 0xA0000079
R0 = 0x00000001 R1 = 0xFFFFFFAC R2 = 0x00000000 R3 = 0x02B6AAB0
R4 = 0x0514C3C8 R5 = 0x00000000 R6 = 0x00000001 R7 = 0x00000014
R8 = 0x778D4B3D R9 = 0x03860C28 R10 = 0x003C0001 R11 = 0x00000000
R12 = 0x30000093 R13 = 0x00110000 R14 = 0x03340000 R15 = 0x04D69DF0
R16 = 0x00000000 R17 = 0x00000000 R18 = 0x04D69E48 R19 = 0x00000001
R20 = 0x0514C3C8 R21 = 0x0000003C R22 = 0x0514C3C8 R23 = 0x00000025
R24 = 0x0514C3C8 R25 = 0x00000025 R26 = 0x003C0001 R27 = 0x73850001
R28 = 0x00000025 R29 = 0x003C0001 R30 = 0x03340000 R31 = 0x02B6AAB0

Stack trace:
PC = 0x01F03708, SP = 0xFFFFFFAC
Frame 00: SP = 0x00000000 PC = 0x00000000

FRNTE-E2-12-CS3750-01 uptime is 2 minutes, 44 seconds

cisco WS-C3750-48P (PowerPC405) processor (revision E0) with 131072K bytes of memory.
Processor board ID CAT0909R05P

Initializing flashfs...

flashfs[1]: 625 files, 10 directories
flashfs[1]: 0 orphaned files, 0 orphaned directories
flashfs[1]: Total bytes: 15998976
flashfs[1]: Bytes used: 15872000
flashfs[1]: Bytes available: 126976
flashfs[1]: flashfs fsck took 5 seconds.
flashfs[1]: Initialization complete....done Initializing flashfs.
Checking for Bootloader upgrade.. not needed

POST: CPU MIC register Tests : Begin
POST: CPU MIC register Tests : End, Status Passed

POST: PortASIC Memory Tests : Begin
POST: PortASIC Memory Tests : End, Status Passed

POST: CPU MIC interface Loopback Tests : Begin
POST: CPU MIC interface Loopback Tests : End, Status Passed

POST: PortASIC RingLoopback Tests : Begin

% There may not be enough space available to collect the complete crashinfo
% It would be advisable to have 282836 bytes free space on flash:crashinfo

Press RETURN to get started!

POST: PortASIC RingLoopback Tests : End, Status Passed

Waiting for Stack Master Election...

POST: Inline Power Controller Tests : Begin
POST: Inline Power Controller Tests : End, Status Passed

POST: PortASIC CAM Subsystem Tests : Begin
POST: PortASIC CAM Subsystem Tests : End, Status Passed

POST: No Cable found on stack port 1
POST: No Cable found on stack port 2

POST: PortASIC Stack Port Loopback Tests : Begin
POST: PortASIC Stack Port Loopback Tests : End, Status Passed

POST: PortASIC Port Loopback Tests : Begin

*Mar 1 00:00:38.403: %STACKMGR-4-SWITCH_ADDED: Switch 1 has been ADDED to the stack

*Mar 1 00:00:51.027: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan1, changed state to down

*Mar 1 00:00:52.370: %SPANTREE-5-EXTENDED_SYSID: Extended SysId enabled for type vlan

*Mar 1 00:00:55: %SYS-6-CLOCKUPDATE: System clock has been updated from 00:00:55 UTC Mon Mar 1 1993 to 01:00:55 MET Mon Mar 1 1993, configured from console by console.

*Mar 1 00:00:55: %SYS-6-CLOCKUPDATE: System clock has been updated from 01:00:55 MET Mon Mar 1 1993 to 01:00:55 MET Mon Mar 1 1993, configured from console by console.

*Mar 1 00:01:26: %SYS-5-CONFIG_I: Configured from memory by console

*Mar 1 00:01:27: %STACKMGR-5-SWITCH_READY: Switch 1 is READY

*Mar 1 00:01:27: %STACKMGR-4-STACK_LINK_CHANGE: Stack Port 1 Switch 1 has changed to state DOWN

*Mar 1 00:01:27: %STACKMGR-4-STACK_LINK_CHANGE: Stack Port 2 Switch 1 has changed to state DOWN

*Mar 1 00:01:28: %SYS-5-RESTART: System restarted --

Cisco IOS Software, C3750 Software (C3750-IPBASEK9-M), Version 12.2(52)SE, RELEASE SOFTWARE (fc3)

Compiled Fri 25-Sep-09 08:32 by sasyamal

*Mar 1 00:01:28: %STACKMGR-5-MASTER_READY: Master Switch 1 is READY

*Mar 1 00:01:28: %SSH-5-ENABLED: SSH 1.99 has been enabled

*Mar 1 00:01:29: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 10.68.248.7 Port 514 started - CLI initiated

*Mar 1 00:01:32: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 10.68.160.50 Port 514 started - CLI initiated

*Mar 1 00:02:28: %PLATFORM-1-CRASHED: System previously crashed with the following message:

*Mar 1 00:02:28: %PLATFORM-1-CRASHED: Cisco IOS Software, C3750 Software (C3750-IPBASEK9-M), Version 12.2(52)SE, RELEASE SOFTWARE (fc3)

*Mar 1 00:02:28: %PLATFORM-1-CRASHED: Compiled Fri 25-Sep-09 08:32 by sasyamal

*Mar 1 00:02:28: %PLATFORM-1-CRASHED:

*Mar 1 00:02:28: %PLATFORM-1-CRASHED: Debug Exception (Could be NULL pointer dereference) Exception (0x2000)!

*Mar 1 00:02:28: %PLATFORM-1-CRASHED:

*Mar 1 00:02:28: %PLATFORM-1-CRASHED: SRR0 = 0x018BA7A0 SRR1 = 0x00029230 SRR2 = 0x017613DC SRR3 = 0x00021000

*Mar 1 00:02:28: %PLATFORM-1-CRASHED: ESR = 0x00000000 DEAR = 0x00000000 TSR = 0x8C000000 DBSR = 0x00800000

*Mar 1 00:02:28: %PLATFORM-1-CRASHED:

*Mar 1 00:02:28: %PLATFORM-1-CRASHED: CPU Register Context:

*Mar 1 00:02:28: %PLATFORM-1-CRASHED: Vector = 0x00002000 PC = 0x01F03708 MSR = 0x00029230 CR = 0x30000095

*Mar 1 00:02:28: %PLATFORM-1-CRASHED: LR = 0x0191B180 CTR = 0x0178AD2C XER = 0xA0000079

*Mar 1 00:02:28: %PLATFORM-1-CRASHED: R0 = 0x00000001 R1 = 0xFFFFFFAC R2 = 0x00000000 R3 = 0x02B6AAB0

*Mar 1 00:02:28: %PLATFORM-1-CRASHED: R4 = 0x0514C3C8 R5 = 0x00000000 R6 = 0x00000001 R7 = 0x00000014

*Mar 1 00:02:28: %PLATFORM-1-CRASHED: R8 = 0x778D4B3D R9 = 0x03860C28 R10 = 0x003C0001 R11 = 0x00000000

*Mar 1 00:02:28: %PLATFORM-1-CRASHED: R12 = 0x30000093 R13 = 0x00110000 R14 = 0x03340000 R15 = 0x04D69DF0

*Mar 1 00:02:28: %PLATFORM-1-CRASHED: R16 = 0x00000000 R17 = 0x00000000 R18 = 0x04D69E48 R19 = 0x00000001

*Mar 1 00:02:28: %PLATFORM-1-CRASHED: R20 = 0x0514C3C8 R21 = 0x0000003C R22 = 0x0514C3C8 R23 = 0x00000025

*Mar 1 00:02:28: %PLATFORM-1-CRASHED: R24 = 0x0514C3C8 R25 = 0x00000025 R26 = 0x003C0001 R27 = 0x73850001

*Mar 1 00:02:28: %PLATFORM-1-CRASHED: R28 = 0x00000025 R29 = 0x003C0001 R30 = 0x03340000 R31 = 0x02B6AAB0

*Mar 1 00:02:28: %PLATFORM-1-CRASHED:

*Mar 1 00:02:28: %PLATFORM-1-CRASHED: Stack trace:

*Mar 1 00:02:28: %PLATFORM-1-CRASHED: PC = 0x01F03708, SP = 0xFFFFFFAC

*Mar 1 00:02:28: %PLATFORM-1-CRASHED: Frame 00: SP = 0x00000000 PC = 0x00000000

*Mar 1 00:02:28: %PLATFORM-1-CRASHED:

*Mar 1 00:02:57: %PARSER-6-VIEW_SWITCH: successfully set to view 'root'.

When I remove the switch of the stack , I have the same issue if I connect a PC.

I don't see warnings for hardware failure.

I have crash info but i don't know how to translate it?

Do you have any idea?

Reinstall the IOS?

Thanks in advance for your help

François

Leo Laohoo · ‎10-08-2013

You've got multiple problems. Here they are:

1. Your switch is undergoing a crash-boot-loop. The switch crashes then tries to create a crashinfo file. For unknown reasons the flash is full so the switch can't write the crashinfo. Because of this the switch "crashes" again, reboots and the cycle repeats itself.

2. You are running 802.1X? With a buggy IOS 12.2(52)SE.

That's it.

To solve your issue, you need to stop the crash-boot-loop. In order for you to do this, you'll need to hold down the mode button while the switch is rebooting and go into ROMmon.

frbachel05 · ‎10-08-2013

Hi Leo,

Thanks for your help.

I deleted files about crash info in the flash to free space and the switch can again write crashinfo.

But the swich is still rebooting...

All my switchs have the IOS version 12.2(52) SE and I didn't have any trouble before this reboot because of this electric cut.

*Mar 1 00:00:32.690: %STACKMGR-4-SWITCH_ADDED: Switch 1 has been ADDED to the stack

*Mar 1 00:00:45.315: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan1, changed state to down

*Mar 1 00:00:46.657: %SPANTREE-5-EXTENDED_SYSID: Extended SysId enabled for type vlan

*Mar 1 00:00:49: %SYS-6-CLOCKUPDATE: System clock has been updated from 00:00:49 UTC Mon Mar 1 1993 to 01:00:49 MET Mon Mar 1 1993, configured from console by console.

*Mar 1 00:00:49: %SYS-6-CLOCKUPDATE: System clock has been updated from 01:00:49 MET Mon Mar 1 1993 to 01:00:49 MET Mon Mar 1 1993, configured from console by console.

*Mar 1 00:01:21: %SYS-5-CONFIG_I: Configured from memory by console

*Mar 1 00:01:21: %STACKMGR-5-SWITCH_READY: Switch 1 is READY

*Mar 1 00:01:21: %STACKMGR-4-STACK_LINK_CHANGE: Stack Port 1 Switch 1 has changed to state DOWN

*Mar 1 00:01:21: %STACKMGR-4-STACK_LINK_CHANGE: Stack Port 2 Switch 1 has changed to state DOWN

*Mar 1 00:01:22: %SYS-5-RESTART: System restarted --

Cisco IOS Software, C3750 Software (C3750-IPBASEK9-M), Version 12.2(52)SE, RELEASE SOFTWARE (fc3)

Compiled Fri 25-Sep-09 08:32 by sasyamal

*Mar 1 00:01:22: %SSH-5-ENABLED: SSH 1.99 has been enabled

*Mar 1 00:01:23: %STACKMGR-5-MASTER_READY: Master Switch 1 is READY

*Mar 1 00:01:23: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 10.68.248.7 Port 514 started - CLI initiated

*Mar 1 00:01:26: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 10.68.160.50 Port 514 started - CLI initiated

*Mar 1 00:02:22: %PLATFORM-1-CRASHED: System previously crashed with the following message:

*Mar 1 00:02:22: %PLATFORM-1-CRASHED: Cisco IOS Software, C3750 Software (C3750-IPBASEK9-M), Version 12.2(52)SE, RELEASE SOFTWARE (fc3)

*Mar 1 00:02:22: %PLATFORM-1-CRASHED: Compiled Fri 25-Sep-09 08:32 by sasyamal

*Mar 1 00:02:22: %PLATFORM-1-CRASHED:

*Mar 1 00:02:22: %PLATFORM-1-CRASHED: Debug Exception (Could be NULL pointer dereference) Exception (0x2000)!

*Mar 1 00:02:22: %PLATFORM-1-CRASHED:

*Mar 1 00:02:22: %PLATFORM-1-CRASHED: SRR0 = 0x02740EAC SRR1 = 0x00029230 SRR2 = 0x017613DC SRR3 = 0x00021000

*Mar 1 00:02:22: %PLATFORM-1-CRASHED: ESR = 0x00000000 DEAR = 0x00000000 TSR = 0x8C000000 DBSR = 0x00800000

*Mar 1 00:02:22: %PLATFORM-1-CRASHED:

*Mar 1 00:02:22: %PLATFORM-1-CRASHED: CPU Register Context:

*Mar 1 00:02:22: %PLATFORM-1-CRASHED: Vector = 0x00002000 PC = 0x01F03708 MSR = 0x00029230 CR = 0x35000095

*Mar 1 00:02:22: %PLATFORM-1-CRASHED: LR = 0x0191B180 CTR = 0x0178AD2C XER = 0xE000007E

*Mar 1 00:02:22: %PLATFORM-1-CRASHED: R0 = 0x00000001 R1 = 0xFFFFFFAC R2 = 0x00000000 R3 = 0x02B6AAB0

*Mar 1 00:02:22: %PLATFORM-1-CRASHED: R4 = 0x0514E15C R5 = 0x00000000 R6 = 0x00000001 R7 = 0x00000014

*Mar 1 00:02:22: %PLATFORM-1-CRASHED: R8 = 0x77CFE935 R9 = 0x03860C28 R10 = 0x003C0001 R11 = 0x00000000

*Mar 1 00:02:22: %PLATFORM-1-CRASHED: R12 = 0x35000093 R13 = 0x00110000 R14 = 0x03340000 R15 = 0x053703A0

*Mar 1 00:02:22: %PLATFORM-1-CRASHED: R16 = 0x00000000 R17 = 0x00000000 R18 = 0x053703F8 R19 = 0x00000001

*Mar 1 00:02:22: %PLATFORM-1-CRASHED: R20 = 0x0514E15C R21 = 0x0000003C R22 = 0x0514E15C R23 = 0x00000034

*Mar 1 00:02:22: %PLATFORM-1-CRASHED: R24 = 0x0514E15C R25 = 0x00000034 R26 = 0x003C0001 R27 = 0x73850001

*Mar 1 00:02:22: %PLATFORM-1-CRASHED: R28 = 0x00000034 R29 = 0x003C0001 R30 = 0x03340000 R31 = 0x02B6AAB0

*Mar 1 00:02:22: %PLATFORM-1-CRASHED:

*Mar 1 00:02:22: %PLATFORM-1-CRASHED: Stack trace:

*Mar 1 00:02:22: %PLATFORM-1-CRASHED: PC = 0x01F03708, SP = 0xFFFFFFAC

*Mar 1 00:02:22: %PLATFORM-1-CRASHED: Frame 00: SP = 0x00000000 PC = 0x00000000

*Mar 1 00:02:22: %PLATFORM-1-CRASHED:

*Mar 1 00:04:18: %PARSER-6-VIEW_SWITCH: successfully set to view 'root'.

Leo Laohoo · ‎10-08-2013

Ok, this is the stack master, right?

Go back to ROMmon and delete all the config.

frbachel05 · ‎10-09-2013

Hi,

No I downgraded the switch to member only

I deleted all the configs and the flash was empty.

I copied a new IOS to the flash in rommon mode with the command "copy xmodem flash:"...
I set boot flash with this IOS.

I configured basic configuration with one interface vlan and one port to make tests

But if i connect a port, the switch is rebooting continuously again ...

I think I need to open call at Cisco...

Thanks