3750G sporadically rebooting

Unanswered Question
Jan 8th, 2010
User Badges:

We have a stack of two 3750G-24TS switches. Both have been operating without fault for months, until earlier today the slave switch rebooted. It has rebooted again several times with apparently no cause to it, and nothing obvious logged when this occurred.


There's no crash info on the flash (unless this has to be turned on?). Is there any obvious debug levels to try turning on?


Any assistance greatly appreciated, thanks!


Output from sh ver:
Cisco IOS Software, C3750 Software (C3750-IPBASE-M), Version 12.2(25)SEB2, RELEASE SOFTWARE (fc1)
Copyright (c) 1986-2005 by Cisco Systems, Inc.
Compiled Wed 08-Jun-05 01:19 by yenanh


ROM: Bootstrap program is C3750 boot loader
BOOTLDR: C3750 Boot Loader (C3750-HBOOT-M) Version 12.1(14r)EA1a, RELEASE SOFTWARE (fc1)


switch2 uptime is 31 weeks, 3 days, 7 hours, 55 minutes
System returned to ROM by power-on
System restarted at 08:11:01 UTC Tue Jun 2 2009
System image file is "flash:c3750-ipbase-mz.122-25.SEB2/c3750-ipbase-mz.122-25.SEB2.bin"


cisco WS-C3750G-24TS (PowerPC405) processor (revision M0) with 118784K/12280K bytes of memory.
Processor board ID CAT1028NJ94
Last reset from power-on
3 Virtual Ethernet interfaces
56 Gigabit Ethernet interfaces
The password-recovery mechanism is enabled.


512K bytes of flash-simulated non-volatile configuration memory.
Base ethernet MAC Address       : 00:18:B9:6F:C2:80
Motherboard assembly number     : 73-7058-13
Power supply part number        : 341-0045-01
Motherboard serial number       : CAT10276MZS
Power supply serial number      : PHI1019L17W
Model revision number           : M0
Motherboard revision number     : C0
Model number                    : WS-C3750G-24TS-S
System serial number            : CAT1028NJ94
Hardware Board Revision Number  : 0x09



Switch   Ports  Model              SW Version              SW Image           
------   -----  -----              ----------              ----------         
     1   28     WS-C3750G-24TS     12.2(25)SEB2            C3750-IPBASE-M     
*    2   28     WS-C3750G-24TS     12.2(25)SEB2            C3750-IPBASE-M    



Switch 01
---------
Switch Uptime                   : 3 hours, 36 minutes
Base ethernet MAC Address       : 00:15:62:CF:9E:80
Motherboard assembly number     : 73-7058-13
Power supply part number        : 341-0045-01
Motherboard serial number       : CAT093605T1
Power supply serial number      : PHI0928L1GJ
Model revision number           : M0
Motherboard revision number     : B0
Model number                    : WS-C3750G-24TS-S
System serial number            : CAT0936N3VT


Configuration register is 0xF

  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 5 (2 ratings)
Loading.
Reza Sharifi Fri, 01/08/2010 - 08:49
User Badges:
  • Super Bronze, 10000 points or more
  • Cisco Designated VIP,

    2017 LAN

Hello Barnaby,


Have you turned on or enabled any feature on the switches lately?  It is really hard to tell specially if there is no crash file available.  You can periodically check the status of the CPU by using the command "sh proce cpu" and memory to see if any specific feature is hugging up your process and memory.  You can also clear some of  the TCP ports by using the command "clea tcp local" that may be taking up a lot proccesses and memory. I also noticed that you are running an older IOS, so it may be time to upgrade to a newer version.



HTH

Reza

barnabyg1 Fri, 01/08/2010 - 09:26
User Badges:

Thanks for the fast response! Genuinely we haven't changed any config at all recently, which is what makes this so strange.


We've cacti monitoring CPU and it shows consistently 6%, memory seems fine too.


Bit hesitent about upgrading IOS, but I guess we don't have many options! Presumably this involves a support contract too.

Reza Sharifi Fri, 01/08/2010 - 09:36
User Badges:
  • Super Bronze, 10000 points or more
  • Cisco Designated VIP,

    2017 LAN

Hi,


I would also open a ticket with TAC to make sure all your bases are covered, but most likly TAC will also suggest an upgrade.


HTH

Reza

sachinraja Fri, 01/08/2010 - 10:47
User Badges:
  • Red, 2250 points or more

Hi Barbany


There are 2 kinds of crashinfo.. You should be getting the basic crashinfo file, once the switch reboots and comes back online.. you can get the crash info file using the more command or using the show tech ... am sure the TAC engineer would ask for the "show tech" command.. Can you get onto the console of the crashed switch and see this ? also get a show version from the console of the switch...


The second kind of crashinfo is called extended crashinfo which gets generated automatically when the switch goes down. you can use the "exception crashinfo" command to enable this.. But this is supported only from 3750 IOS versio 12.2(25)SEC or later, which means you need to upgrade your IOS...


Hope this helpls.. all the best


Raj

barnabyg1 Fri, 01/08/2010 - 11:09
User Badges:

Hi - you say console of the crashed switch  - I can get into the console for the stack (presumably the master), not sure how to get onto a console of a specific switch in the stack (the single slave in this case)?


I've done dir flash:, dir flash1: and dir flash2: and none show any crash info files.


I guessing potentially the problem is hardware - if say the power supply was failing, would no crash info be synonymous with power just being cut?


thanks,


Barnaby

sachinraja Fri, 01/08/2010 - 11:16
User Badges:
  • Red, 2250 points or more

Yeps.. Could be a power issue as well..


did you try getting onto the failed stack member using "session stack-member-number " from the master switch ? Show version given by you still shows the entire switch and not the boot reason for that stack member.. if you see the reload reason - as power boot, you can confirm that it was powered up manually or due to power issues...


Regards

Raj

barnabyg1 Fri, 01/08/2010 - 11:23
User Badges:

Thanks for the tip.


This is the output, so sounds like you're right. I'll get a body out to the data centre on Monday to replace power cables, etc. Let's hope it's the most obvious thing! :-)


cheers.


switch2-1#sh ver

Cisco IOS Software, C3750 Software (C3750-IPBASE-M), Version 12.2(25)SEB2, RELEASE SOFTWARE (fc1)

Copyright (c) 1986-2005 by Cisco Systems, Inc.

Compiled Wed 08-Jun-05 01:19 by yenanh


ROM: Bootstrap program is C3750 boot loader

BOOTLDR: C3750 Boot Loader (C3750-HBOOT-M) Version 12.1(14r)EA1a, RELEASE SOFTWARE (fc1)


switch2-1 uptime is 6 hours, 46 minutes

System returned to ROM by power-on

System restarted at 12:34:22 UTC Fri Jan 8 2010

System image file is "flash:c3750-ipbase-mz.122-25.SEB2/c3750-ipbase-mz.122-25.SEB2.bin"


cisco WS-C3750G-24TS (PowerPC405) processor (revision M0) with 118784K/12280K bytes of memory.

Processor board ID CAT0936N3VT

Last reset from power-on

3 Virtual Ethernet interfaces

56 Gigabit Ethernet interfaces

The password-recovery mechanism is enabled.


512K bytes of flash-simulated non-volatile configuration memory.

Base ethernet MAC Address       : 00:15:62:CF:9E:80

Motherboard assembly number     : 73-7058-13

Power supply part number        : 341-0045-01

Motherboard serial number       : CAT093605T1

Power supply serial number      : PHI0928L1GJ

Model revision number           : M0

Motherboard revision number     : B0

Model number                    : WS-C3750G-24TS-S

System serial number            : CAT0936N3VT

Hardware Board Revision Number  : 0x09


Configuration register is 0xF

sachinraja Fri, 01/08/2010 - 11:28
User Badges:
  • Red, 2250 points or more

Cool Barnaby...


It looks obvious now that it is due to power-on .. somebody might have just disrupted the cable when doing someother activity This has happened many a times due to messy cabling in datacenters.. You are right, replace the power cable and monitor the switch... Hope this helps.. all the best. rate replies if found useful..


Thanks

Raj

barnabyg1 Fri, 01/08/2010 - 11:29
User Badges:

Just trying to recall from memory - is it a standard kettle lead (as we call then in the UK) to the power supply on the switch?

sachinraja Fri, 01/08/2010 - 11:33
User Badges:
  • Red, 2250 points or more

Yes Barnaby.. Attaching the picture of 3750 with the cables associated.. Just googled

"standard kettle lead" and saw the picture.. it looks right..


Raj

Attachment: 
barnabyg1 Fri, 01/08/2010 - 11:36
User Badges:

Sweet. Thanks a lot, you've been very helpful.

Actions

This Discussion