WS-C3750G-24TS-1U stack crashed IOS ver: 15.0(1)SE

rizwanr74 · ‎11-03-2011

Gentlemen,

I have a situation. I am running two-switch 3750-stack and it crashed this morning. IOS version the switches is: "flash:c3750-ipservicesk9-mz.150-1.SE.bin"

Any help is appreciated, please see the log below...

--------------------------------------------------------------------------------

SRPRSRGBYA#

Log Buffer (10000 bytes):

3:40.840: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan104, changed state to up

Nov 3 13:13:40.840: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan201, changed state to up

Nov 3 13:13:40.840: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan202, changed state to up

Nov 3 13:13:40.840: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan203, changed state to up

Nov 3 13:13:40.840: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan204, changed state to up

Nov 3 13:13:40.840: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan233, changed state to up

Nov 3 13:13:40.840: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan900, changed state to up

Nov 3 13:13:40.840: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan901, changed state to up

Nov 3 13:13:40.840: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan902, changed state to up

Nov 3 13:13:41.595: %STACKMGR-5-MASTER_READY: Master Switch 2 is READY

Nov 3 13:13:41.813: %DUAL-5-NBRCHANGE: EIGRP-IPv4 900: Neighbor 10.14.0.9 (Vlan233) is up: new adjacency

Nov 3 13:13:41.822: %LINEPROTO-5-UPDOWN: Line protocol on Interface Port-channel2, changed state to up

Nov 3 13:13:41.822: %LINEPROTO-5-UPDOWN: Line protocol on Interface Port-channel3, changed state to up

Nov 3 13:13:41.822: %LINEPROTO-5-UPDOWN: Line protocol on Interface Port-channel4, changed state to up

Nov 3 13:13:41.822: %LINEPROTO-5-UPDOWN: Line protocol on Interface Port-channel5, changed state to up

Nov 3 13:13:41.822: %LINEPROTO-5-UPDOWN: Line protocol on Interface Port-channel25, changed state to up

Nov 3 13:13:50.160: %OSPF-5-ADJCHG: Process 49, Nbr 10.13.0.13 on GigabitEthernet2/0/1 from LOADING to FULL, Loading Done

Nov 3 13:14:14.781: %DUAL-5-NBRCHANGE: EIGRP-IPv4 900: Neighbor 10.14.0.9 (Vlan233) is down: holding time expired

Nov 3 13:14:34.418: %DUAL-5-NBRCHANGE: EIGRP-IPv4 900: Neighbor 10.14.0.9 (Vlan233) is up: new adjacency

Nov 3 13:16:04.889: %STACKMGR-4-STACK_LINK_CHANGE: Stack Port 1 Switch 2 has changed to state UP

Nov 3 13:16:04.889: %STACKMGR-4-STACK_LINK_CHANGE: Stack Port 2 Switch 2 has changed to state UP

Nov 3 13:17:02.888: %STACKMGR-4-SWITCH_ADDED: Switch 1 has been ADDED to the stack

Nov 3 13:17:13.022: %STACKMGR-5-SWITCH_READY: Switch 1 is READY

Nov 3 13:17:13.022: %STACKMGR-4-STACK_LINK_CHANGE: Stack Port 1 Switch 1 has changed to state UP

Nov 3 13:17:13.022: %STACKMGR-4-STACK_LINK_CHANGE: Stack Port 2 Switch 1 has changed to state UP

*Mar 1 00:02:12.372: %STACKMGR-4-SWITCH_ADDED: Switch 1 has been ADDED to the stack (Switch-1)

*Mar 1 00:02:12.372: %STACKMGR-4-SWITCH_ADDED: Switch 2 has been ADDED to the stack (Switch-1)

Auth Manager registration failed (Switch-1)

(Switch-1)

*Mar 1 00:02:16.390: %SPANTREE-5-EXTENDED_SYSID: Extended SysId enabled for type vlan (SRPRSRGBYA-1)

*Mar 1 00:02:17.673: %DC-4-FILE_OPEN_WARNING: Not able to open flash:/dc_profile_dir/dc_default_profiles.txt (SRPRSRGBYA-1)

*Mar 1 00:02:17.673: %DC-6-DEFAULT_INIT_INFO: Default Profiles DB not loaded. (SRPRSRGBYA-1)

*Nov 3 13:17:11.529: %SYS-6-CLOCKUPDATE: System clock has been updated from 13:17:11 UTC Thu Nov 3 2011 to 08:17:11 EST Thu Nov 3 2011, configured from console by console. (SRPRSRGBYA-1)

*Nov 3 13:17:11.529: %SYS-6-CLOCKUPDATE: System clock has been updated from 08:17:11 EST Thu Nov 3 2011 to 09:17:11 EDT Thu Nov 3 2011, configured from console by console. (SRPRSRGBYA-1)

*Nov 3 13:17:12.284: %SW_VLAN-6-VTP_DOMAIN_NAME_CHG: VTP domain name changed to Null. (SRPRSRGBYA-1)

*Nov 3 13:17:12.451: %SYS-5-CONFIG_I: Configured from memory by console (SRPRSRGBYA-1)

*Nov 3 13:17:12.661: %STACKMGR-5-SWITCH_READY: Switch 2 is READY (SRPRSRGBYA-1)

*Nov 3 13:17:12.787: %LINEPROTO-5-UPDOWN: Line protocol on Interface Loopback0, changed state to up (SRPRSRGBYA-1)

*Nov 3 13:17:13.013: %STACKMGR-5-MASTER_READY: Master Switch 2 is READY (SRPRSRGBYA-1)

*Nov 3 13:17:13.198: %SYS-5-RESTART: System restarted -- (SRPRSRGBYA-1)

Cisco IOS Software, C3750 Software (C3750-IPSERVICESK9-M), Version 15.0(1)SE, RELEASE SOFTWARE (fc1) (SRPRSRGBYA-1)

Technical Support:

http://www.cisco.com/techsupport

(SRPRSRGBYA-1)

Nov 3 13:17:17.468: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/25, changed state to up

Nov 3 13:17:18.298: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/3, changed state to up

Nov 3 13:17:18.474: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/25, changed state to up

Nov 3 13:17:18.667: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/1, changed state to up

Nov 3 13:17:18.676: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/4, changed state to up

Nov 3 13:17:18.801: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/2, changed state to up

Nov 3 13:17:18.801: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/5, changed state to up

Nov 3 13:17:18.835: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/14, changed state to up

Nov 3 13:17:18.877: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/6, changed state to up

Nov 3 13:17:18.986: %LINK-3-UPDOWN: Interface Port-channel6, changed state to up

Nov 3 13:17:19.305: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/3, changed state to up

Nov 3 13:17:19.674: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/1, changed state to up

Nov 3 13:17:19.682: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/4, changed state to up

Nov 3 13:17:19.808: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/2, changed state to up

Nov 3 13:17:19.808: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/5, changed state to up

Nov 3 13:17:19.842: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/14, changed state to up

Nov 3 13:17:19.884: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/6, changed state to up

Nov 3 13:17:19.993: %LINEPROTO-5-UPDOWN: Line protocol on Interface Port-channel6, changed state to up

Nov 3 13:18:04.620: %OSPF-5-ADJCHG: Process 49, Nbr 10.13.0.12 on GigabitEthernet1/0/1 from LOADING to FULL, Loading Done

Nov 3 13:18:13.336: %PLATFORM-1-CRASHED: System previously crashed with the following message: (SRPRSRGBYA-1)

Nov 3 13:18:13.336: %PLATFORM-1-CRASHED: Cisco IOS Software, C3750 Software (C3750-IPSERVICESK9-M), Version 15.0(1)SE, RELEASE SOFTWARE (fc1) (SRPRSRGBYA-1)

Nov 3 13:18:13.336: %PLATFORM-1-CRASHED: Technical Support:

http://www.cisco.com/techsupport

(SRPRSRGBYA-1)

Nov 3 13:18:13.336: %PLATFORM-1-CRASHED: Compiled Wed 20-Jul-11 09:32 by prod_rel_team (SRPRSRGBYA-1)

Nov 3 13:18:13.336: %PLATFORM-1-CRASHED: (SRPRSRGBYA-1)

Nov 3 13:18:13.336: %PLATFORM-1-CRASH

Nov 3 13:18:13.336: %PLATFORM-1-CRASHED: ESR: 0x00000000 (SRPRSRGBYA-1)

Nov 3 13:18:13.336: %PLATFORM-1-CRASHED: SRR0: 0x013D176C SRR1: 0x00029230 SRR2: 0x032FA4FC SRR3: 0x00029230 (SRPRSRGBYA-1)

Nov 3 13:18:13.336: %PLATFORM-1-CRASHED: BESRA: 0x00000000, BESRB: 0x00000000, BEAR: 0x00000000 (SRPRSRGBYA-1)

Nov 3 13:18:13.336: %PLATFORM-1-CRASHED: (SRPRSRGBYA-1)

Nov 3 13:18:13.336: %PLATFORM-1-CRASHED: CPU Register Context: (SRPRSRGBYA-1)

Nov 3 13:18:13.336: %PLATFORM-1-CRASHED: Vector = 0x00000200 PC = 0x032FA4FC MSR = 0x00029230 CR = 0x53005095 (SRPRSRGBYA-1)

Nov 3 13:18:13.336: %PLATFORM-1-CRASHED: LR = 0x032FA4F0 CTR = 0x032B229C XER = 0xE000001F (SRPRSRGBYA

Nov 3 13:18:13.336: %PLATFORM-1-CRASHED: R0 = 0x00000000 R1 = 0x00FCE2E8 R2 = 0x00000000 R3 = 0x00000000 (SRPRSRGBYA-1)

Nov 3 13:18:13.336: %PLATFORM-1-CRASHED: R4 = 0x023165A8 R5 = 0x00000000 R6 = 0x0330A460 R7 = 0x00FCE3A8 (SRPRSRGBYA-1)

Nov 3 13:18:13.336: %PLATFORM-1-CRASHED: R8 = 0x0456B238 R9 = 0x15A3C78B R10 = 0x00000000 R11 = 0x00FCE2D8 (SRPRSRGBYA-1)

Nov 3 13:18:13.336: %PLATFORM-1-CRASHED: R12 = 0x33005093 R13 = 0x00110000 R14 = 0x00000000 R15 = 0x00000001 (SRPRSRGBYA-1)

Nov 3 13:18:13.336: %PLATFORM-1-CRASHED: R16 = 0xFFFFFFFF R17 = 0x00000000 R18 = 0x00000000 R19 = 0x00000000 (SRPRSRGBYA-1)

Nov 3 13:18:13.336: %PLATFORM-1-CRASHED: R20 = 0x00

Nov 3 13:18:13.336: %PLATFORM-1-CRASHED: R24 = 0x00000000 R25 = 0x00000000 R26 = 0x00000000 R27 = 0x04C9253C (SRPRSRGBYA-1)

Nov 3 13:18:13.336: %PLATFORM-1-CRASHED: R28 = 0x023165A8 R29 = 0x6D617874 R30 = 0x00000001 R31 = 0x6D617874 (SRPRSRGBYA-1)

Nov 3 13:18:13.336: %PLATFORM-1-CRASHED: (SRPRSRGBYA-1)

Nov 3 13:18:13.336: %PLATFORM-1-CRASHED: Stack trace: (SRPRSRGBYA-1)

Nov 3 13:18:13.336: %PLATFORM-1-CRASHED: PC = 0x032FA4FC, SP = 0x00FCE2E8 (SRPRSRGBYA-1)

Nov 3 13:18:13.336: %PLATFORM-1-CRASHED: Frame 00: SP = 0x00FCE320 PC = 0x032FA4AC (SRPRSRGBYA-1)

Nov 3 13:18:13.336: %PLATFORM-1-CRASHED: Frame 01: SP = 0x00FCE340 PC = 0x036F10D8 (SRPRSRGBYA-1)

Nov

Nov 3 13:18:13.336: %PLATFORM-1-CRASHED: Frame 03: SP = 0x00FCE3B0 PC = 0x023167D4 (SRPRSRGBYA-1)

Nov 3 13:18:13.336: %PLATFORM-1-CRASHED: Frame 04: SP = 0x00FCE3C0 PC = 0x02365AC8 (SRPRSRGBYA-1)

Nov 3 13:18:13.336: %PLATFORM-1-CRASHED: Frame 05: SP = 0x00FCE4E8 PC = 0x02368540 (SRPRSRGBYA-1)

Nov 3 13:18:13.336: %PLATFORM-1-CRASHED: Frame 06: SP = 0x00FCE508 PC = 0x0238AC1C (SRPRSRGBYA-1)

Nov 3 13:18:13.336: %PLATFORM-1-CRASHED: Frame 07: SP = 0x00FCE540 PC = 0x0238B1B8 (SRPRSRGBYA-1)

Nov 3 13:18:13.336: %PLATFORM-1-CRASHED: Frame 08: SP = 0x00FCE558 PC = 0x0238B6C8 (SRPRSRGBYA-1)

Nov 3 13:18:13.336: %PLATFORM-1-CRASHED: Frame 09: SP = 0x00FCE570

Nov 3 13:18:13.336: %PLATFORM-1-CRASHED: Frame 10: SP = 0x00FCE610 PC = 0x0237A4FC (SRPRSRGBYA-1)

Nov 3 13:18:13.336: %PLATFORM-1-CRASHED: Frame 11: SP = 0x00FCE618 PC = 0x01CB3E68 (SRPRSRGBYA-1)

Nov 3 13:18:13.336: %PLATFORM-1-CRASHED: Frame 12: SP = 0x00000000 PC = 0x01CADCDC (SRPRSRGBYA-1)

Nov 3 13:18:13.336: %PLATFORM-1-CRASHED: (SRPRSRGBYA-1)

SRPRSRGBYA#

SRPRSRGBYA#show version

Cisco IOS Software, C3750 Software (C3750-IPSERVICESK9-M), Version 15.0(1)SE, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2011 by Cisco Systems, Inc.
Compiled Wed 20-Jul-11 09:32 by prod_rel_team

ROM: Bootstrap program is C3750 boot loader
BOOTLDR: C3750 Boot Loader (C3750-HBOOT-M) Version 12.2(44)SE5, RELEASE SOFTWARE (fc1)

SRPRSRGBYA uptime is 5 weeks, 6 days, 7 hours, 38 minutes
System returned to ROM by power-on
System restarted at 04:08:42 EDT Fri Sep 23 2011
System image file is "flash:c3750-ipservicesk9-mz.150-1.SE.bin"

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

cisco WS-C3750G-24TS-1U (PowerPC405) processor (revision C0) with 131072K bytes of memory.
Processor board ID FOC0932U025
Last reset from power-on
16 Virtual Ethernet interfaces
56 Gigabit Ethernet interfaces
The password-recovery mechanism is enabled.

512K bytes of flash-simulated non-volatile configuration memory.
Base ethernet MAC Address       : 00:15:2B:11:BC:80
Motherboard assembly number     : 73-9637-08
Power supply part number        : 341-0098-01
Motherboard serial number       : FOC09312SCA
Power supply serial number      : DCA09250N0Z
Model revision number           : C0
Motherboard revision number     : A0
Model number                    : WS-C3750G-24TS-S1U
System serial number            : FOC0932U025
SFP Module assembly part number : 73-7757-03
SFP Module revision Number      : A0
SFP Module serial number        : CAT092815Z8
Top Assembly Part Number        : 800-25730-02
Top Assembly Revision Number    : A0
Version ID                      : 02
CLEI Code Number                : CNMWS00ARB
Hardware Board Revision Number : 0x05

Switch Ports Model              SW Version            SW Image
------ ----- -----              ----------            ----------
     1 28    WS-C3750G-24TS-1U 15.0(1)SE             C3750-IPSERVICESK9-M
*    2 28    WS-C3750G-24TS-1U 15.0(1)SE             C3750-IPSERVICESK9-M

Switch 01
---------
Switch Uptime                   : 2 hours, 31 minutes
Base ethernet MAC Address       : 00:14:1C:B5:F4:80
Motherboard assembly number     : 73-9637-08
Power supply part number        : 341-0098-01
Motherboard serial number       : FOC092031FW
Power supply serial number      : DCA09170AVT
Model revision number           : C0
Motherboard revision number     : A0
Model number                    : WS-C3750G-24TS-S1U
System serial number            : FOC0920U0FN
SFP Module assembly part number : 73-7757-02
SFP Module revision number      : A0
SFP Module serial number        : CAT091705LV
Top assembly part number        : 800-25730-02
Top assembly revision number    : A0
Version ID                      : 02
CLEI Code Number                : CNMWS00ARB

Configuration register is 0xF

SRPRSRGBYA#

Leo Laohoo · ‎11-03-2011

I'm no big fan of running 15.0(1)SE on 3750. I'd recommend either 12.2(55)SE4 or 12.2(58)SE2 or later.

15.0(1)SE1 has a memory leak issue.

fb_webuser · ‎11-03-2011

did you investigate the possibility of a power failure ?

---

Posted by WebUser Ahmed Rasmy

rizwanr74 · ‎11-04-2011

Hi Web User,

Yes, did probe into see whether there was any power failure but there was none.

The master switch crashed while I was in the config-mode configuring Tacace.