01-08-2010 08:15 AM - edited 03-06-2019 09:12 AM
We have a stack of two 3750G-24TS switches. Both have been operating without fault for months, until earlier today the slave switch rebooted. It has rebooted again several times with apparently no cause to it, and nothing obvious logged when this occurred.
There's no crash info on the flash (unless this has to be turned on?). Is there any obvious debug levels to try turning on?
Any assistance greatly appreciated, thanks!
Output from sh ver:
Cisco IOS Software, C3750 Software (C3750-IPBASE-M), Version 12.2(25)SEB2, RELEASE SOFTWARE (fc1)
Copyright (c) 1986-2005 by Cisco Systems, Inc.
Compiled Wed 08-Jun-05 01:19 by yenanh
ROM: Bootstrap program is C3750 boot loader
BOOTLDR: C3750 Boot Loader (C3750-HBOOT-M) Version 12.1(14r)EA1a, RELEASE SOFTWARE (fc1)
switch2 uptime is 31 weeks, 3 days, 7 hours, 55 minutes
System returned to ROM by power-on
System restarted at 08:11:01 UTC Tue Jun 2 2009
System image file is "flash:c3750-ipbase-mz.122-25.SEB2/c3750-ipbase-mz.122-25.SEB2.bin"
cisco WS-C3750G-24TS (PowerPC405) processor (revision M0) with 118784K/12280K bytes of memory.
Processor board ID CAT1028NJ94
Last reset from power-on
3 Virtual Ethernet interfaces
56 Gigabit Ethernet interfaces
The password-recovery mechanism is enabled.
512K bytes of flash-simulated non-volatile configuration memory.
Base ethernet MAC Address : 00:18:B9:6F:C2:80
Motherboard assembly number : 73-7058-13
Power supply part number : 341-0045-01
Motherboard serial number : CAT10276MZS
Power supply serial number : PHI1019L17W
Model revision number : M0
Motherboard revision number : C0
Model number : WS-C3750G-24TS-S
System serial number : CAT1028NJ94
Hardware Board Revision Number : 0x09
Switch Ports Model SW Version SW Image
------ ----- ----- ---------- ----------
1 28 WS-C3750G-24TS 12.2(25)SEB2 C3750-IPBASE-M
* 2 28 WS-C3750G-24TS 12.2(25)SEB2 C3750-IPBASE-M
Switch 01
---------
Switch Uptime : 3 hours, 36 minutes
Base ethernet MAC Address : 00:15:62:CF:9E:80
Motherboard assembly number : 73-7058-13
Power supply part number : 341-0045-01
Motherboard serial number : CAT093605T1
Power supply serial number : PHI0928L1GJ
Model revision number : M0
Motherboard revision number : B0
Model number : WS-C3750G-24TS-S
System serial number : CAT0936N3VT
Configuration register is 0xF
01-08-2010 08:49 AM
Hello Barnaby,
Have you turned on or enabled any feature on the switches lately? It is really hard to tell specially if there is no crash file available. You can periodically check the status of the CPU by using the command "sh proce cpu" and memory to see if any specific feature is hugging up your process and memory. You can also clear some of the TCP ports by using the command "clea tcp local" that may be taking up a lot proccesses and memory. I also noticed that you are running an older IOS, so it may be time to upgrade to a newer version.
HTH
Reza
01-08-2010 09:26 AM
Thanks for the fast response! Genuinely we haven't changed any config at all recently, which is what makes this so strange.
We've cacti monitoring CPU and it shows consistently 6%, memory seems fine too.
Bit hesitent about upgrading IOS, but I guess we don't have many options! Presumably this involves a support contract too.
01-08-2010 09:36 AM
Hi,
I would also open a ticket with TAC to make sure all your bases are covered, but most likly TAC will also suggest an upgrade.
HTH
Reza
01-08-2010 10:47 AM
Hi Barbany
There are 2 kinds of crashinfo.. You should be getting the basic crashinfo file, once the switch reboots and comes back online.. you can get the crash info file using the more command or using the show tech ... am sure the TAC engineer would ask for the "show tech" command.. Can you get onto the console of the crashed switch and see this ? also get a show version from the console of the switch...
The second kind of crashinfo is called extended crashinfo which gets generated automatically when the switch goes down. you can use the "exception crashinfo" command to enable this.. But this is supported only from 3750 IOS versio 12.2(25)SEC or later, which means you need to upgrade your IOS...
Hope this helpls.. all the best
Raj
01-08-2010 11:09 AM
Hi - you say console of the crashed switch - I can get into the console for the stack (presumably the master), not sure how to get onto a console of a specific switch in the stack (the single slave in this case)?
I've done dir flash:, dir flash1: and dir flash2: and none show any crash info files.
I guessing potentially the problem is hardware - if say the power supply was failing, would no crash info be synonymous with power just being cut?
thanks,
Barnaby
01-08-2010 11:16 AM
Yeps.. Could be a power issue as well..
did you try getting onto the failed stack member using "session stack-member-number " from the master switch ? Show version given by you still shows the entire switch and not the boot reason for that stack member.. if you see the reload reason - as power boot, you can confirm that it was powered up manually or due to power issues...
Regards
Raj
01-08-2010 11:23 AM
Thanks for the tip.
This is the output, so sounds like you're right. I'll get a body out to the data centre on Monday to replace power cables, etc. Let's hope it's the most obvious thing! :-)
cheers.
switch2-1#sh ver
Cisco IOS Software, C3750 Software (C3750-IPBASE-M), Version 12.2(25)SEB2, RELEASE SOFTWARE (fc1)
Copyright (c) 1986-2005 by Cisco Systems, Inc.
Compiled Wed 08-Jun-05 01:19 by yenanh
ROM: Bootstrap program is C3750 boot loader
BOOTLDR: C3750 Boot Loader (C3750-HBOOT-M) Version 12.1(14r)EA1a, RELEASE SOFTWARE (fc1)
switch2-1 uptime is 6 hours, 46 minutes
System returned to ROM by power-on
System restarted at 12:34:22 UTC Fri Jan 8 2010
System image file is "flash:c3750-ipbase-mz.122-25.SEB2/c3750-ipbase-mz.122-25.SEB2.bin"
cisco WS-C3750G-24TS (PowerPC405) processor (revision M0) with 118784K/12280K bytes of memory.
Processor board ID CAT0936N3VT
Last reset from power-on
3 Virtual Ethernet interfaces
56 Gigabit Ethernet interfaces
The password-recovery mechanism is enabled.
512K bytes of flash-simulated non-volatile configuration memory.
Base ethernet MAC Address : 00:15:62:CF:9E:80
Motherboard assembly number : 73-7058-13
Power supply part number : 341-0045-01
Motherboard serial number : CAT093605T1
Power supply serial number : PHI0928L1GJ
Model revision number : M0
Motherboard revision number : B0
Model number : WS-C3750G-24TS-S
System serial number : CAT0936N3VT
Hardware Board Revision Number : 0x09
Configuration register is 0xF
01-08-2010 11:28 AM
Cool Barnaby...
It looks obvious now that it is due to power-on .. somebody might have just disrupted the cable when doing someother activity This has happened many a times due to messy cabling in datacenters.. You are right, replace the power cable and monitor the switch... Hope this helps.. all the best. rate replies if found useful..
Thanks
Raj
01-08-2010 11:29 AM
Just trying to recall from memory - is it a standard kettle lead (as we call then in the UK) to the power supply on the switch?
01-08-2010 11:33 AM
01-08-2010 11:36 AM
Sweet. Thanks a lot, you've been very helpful.
Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: