Cisco C4503-E is being reloaded without a reason

Mohamed_Saad · ‎03-28-2015

We have a Cisco 4500 switches and some of them is being reloaded without a known reason, Please check the log output and let me know what should be the cause for that ?

Note that we have Cisco LMS managing the network switches.

***********************************

SW#sh ver
Cisco IOS Software, Catalyst 4500 L3 Switch Software (cat4500e-LANBASEK9-M), Version 15.1(2)SG5, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2014 by Cisco Systems, Inc.
Compiled Mon 03-Nov-14 09:04 by prod_rel_team

ROM: 12.2(44r)SG5
SW uptime is 1 hour, 42 minutes
System returned to ROM by reload
System restarted at 16:43:51 GMT Sat Mar 28 2015
System image file is "bootflash:cat4500e-lanbasek9-mz.151-2.SG5.bin"
Darkside Revision 4, Nexu Revision 9, Fortooine Revision 1.40

Last reload reason: reload

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

cisco WS-C4503-E (MPC8548) processor (revision 8) with 524288K bytes of memory.
Processor board ID SPE135100L1
MPC8548 CPU at 1GHz, Supervisor 6L-E
Last reset from Reload
2 Virtual Ethernet interfaces
52 Gigabit Ethernet interfaces
2 Ten Gigabit Ethernet interfaces
511K bytes of non-volatile configuration memory.

Configuration register is 0x2101

**********************************************************

SW#sh platform crashdump
========= Start of Crashinfo Collection ======================

Current Time: 3/28/2015 19:7:56

Last Power Failure: 01/21/2010 08:23:28
Last Reload Status: 00000000
Last Software Reset State: 00000201

========= End of Crashinfo Collection ======================

******************************************************************************************

SW#sh environment status
Power                                             Fan      Inline
Supply Model No          Type       Status       Sensor   Status
------ ---------------- --------- ----------- ------- -------
PS1     PWR-C45-1300ACV   AC 1300W   good         good     good
PS2     none              -- --      --           --       --

Power supplies needed by system : 1
Power supplies currently available : 1

Chassis Type : WS-C4503-E

Power consumed by backplane : 0 Watts

Switch Bandwidth Utilization : 0%

Supervisor Led Color : Green

Module 1 Status Led Color : Green
Module 2 Status Led Color : Green PoE Led Color : Green

Fantray : Good
Fantray removal timeout : 100

**********************************************************************

*Mar 28 07:55:39.295: %SPANTREE-5-EXTENDED_SYSID: Extended SysId enabled for type vlan
*Mar 28 08:55:41.471: %SYS-6-CLOCKUPDATE: System clock has been updated from 07:55:41 UTC Sat Mar 28 2015 to 08:55:41 GMT Sat Mar 28 2015, configured from console by console.
*Mar 28 08:55:41.471: %SYS-6-CLOCKUPDATE: System clock has been updated from 08:55:41 GMT Sat Mar 28 2015 to 08:55:41 GMT Sat Mar 28 2015, configured from console by console.
*Mar 28 08:55:41.803: %C4K_IOSMODPORTMAN-6-MODULEONLINE: Module 1 (WS-X45-SUP6L-E S/N: JAE14020672 Hw: 1.0) is online
*Mar 28 08:55:41.803: %C4K_IOSMODPORTMAN-6-MODULEONLINE: Module 2 (WS-X4648-RJ45V+E S/N: JAE14020HCM Hw: 2.0) is online
*Mar 28 08:55:42.335: %SW_VLAN-6-VTP_DOMAIN_NAME_CHG: VTP domain name changed to LG.
*Mar 28 08:55:42.387: %SYS-5-CONFIG_I: Configured from memory by console
*Mar 28 08:55:42.551: %SYS-5-RESTART: System restarted --
Cisco IOS Software, Catalyst 4500 L3 Switch Software (cat4500e-LANBASEK9-M), Version 15.1(2)SG5, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2014 by Cisco Systems, Inc.
Compiled Mon 03-Nov-14 09:04 by prod_rel_team
*Mar 28 08:55:42.639: %SNMP-5-COLDSTART: SNMP agent on host SW is undergoing a cold start
*Mar 28 08:55:42.699: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 20.199.30.37 port 514 started - CLI initiated
*Mar 28 08:55:45.707: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 72.22.66.1 port 514 started - CLI initiated
*Mar 28 08:55:46.443: Port Te1/1: TwinGig Converter inserted: vendor: CISCO SYSTEMS, p/n: 800-27645-02, s/n: FDO13480NT0
*Mar 28 08:55:46.443: Port Te1/2: TwinGig Converter inserted: vendor: CISCO SYSTEMS, p/n: 800-27645-02, s/n: FDO13480NNM
*Mar 28 08:55:48.739: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 72.25.16.25 port 514 started - CLI initiated

**********************************

Charles Hill · ‎03-28-2015

I ran the show ver and show platform crash through Cisco's output interperter and it showed

INFO: The system has been manually rebooted using "reload" command. This is
not a system crash.

I do not see a bug that is affecting the relase 15.1(2SG5).

If you will do a "show tech" and upload it, I'll be glad to look at it.

Mohamed_Saad · ‎03-28-2015

OK, thanks for the quick reply .. here it is the show tech- support

Charles Hill · ‎03-28-2015

Looks like there may be a memory leak, however I'm not seeing an error indicating it reloaded due to a memory error.

Do a show process memory several times within the next day or so and verify if the memory utilization continues to increase.

WARNING: The following processes are currently holding more than 1 MB of memory
with 'Freed' memory less that 'Holding' memory:
  'GaliosObflLogger' (Holding 2206392 bytes)
This is considered to be high. High memory usage can be normal, or can indicate
a memory leak. To determine this, it is important to see if memory usage changes
over time.
INFO: A memory leak occurs when a process requests or allocates memory and then
forgets to free (de-allocate) the memory when it is finished with that task. As
a result, the memory block is reserved until the router is reloaded. Over time,
more and more memory blocks are allocated by that process until there is no free
memory available. At this point, the router will issue the 'SYS-2-MALLOCFAIL' error
message.

TRY THIS: Analyze the 'show process memory' output for this router over a period
of time (for example, every few hours or days depending on whether you have a fast
or slow leak).
Check to see if memory utilization for the affected process(es) continues to increase
and the amount of freed memory remains the same. The rate at which free memory
disappears depends on how often the event occurs that leads to the leak. A memory
leak is a complex condition sometimes requiring an IOS upgrade to correct. If the
above is in fact occurring, and you are uncertain about how to proceed, use the
Service Request Tool to contact the Cisco TAC for further assistance.

Mohamed_Saad · ‎03-28-2015

Thanks Cehill,

I will run the command tomorrow during the day when there is a lot of traffic going on the switch and see the results.

What do we need to do to fix such issue if it turns out to be memory leaks ?

Charles Hill · ‎03-30-2015

Hey Mohamed,

Upgrade or downgrade can fix a memory leak. Still not sure if this is a memory leak.

Cisco's preferred ios for this platform(gold star beside it) is
15.0.2-SG9(MD).

Take a look at this Caveat/Bug.

Open Caveats for Cisco IOS Release 15.1(2)SG5

CSCum58148

Switch crashes when you modify policer rates of QoS policy..

workaround:

Remove the policy from the targets to which it is attached, modify the policy, and re-attach.

Mohamed_Saad · ‎03-30-2015

Hi cehill,

I have ran the Show Processes Memory couple of times today and the output seems similar 47% usage .

Regarding the upgrade or downgrade I will consider it but not now as I have opened TAC case with Cisco and waiting for analysis.

Charles Hill · ‎03-30-2015

Ok, thanks for the update.

Mohamed_Saad · ‎03-31-2015

Hi,

I have been informed by Cisco that they have a hardware bug CSCtf85481

the Sup Engines will be replaced..all the switches that fails under that hardware revision no. !!

Thanks for your help so far

Charles Hill · ‎03-31-2015

Your welcome.

Thanks for the update.