Switch (WS-C6509-E) module(WS-X6724-SFP) restarts intermittentely

amardram123 · ‎04-02-2010

Hi friends,

I am having WS-X6724-SFP in our 6509, which is getting restarted 2-3 time in a day.. No crashinfo found in show tech.. pasted below some logs relevent to this module,... Need expert suggestion as it will badly impact if does not get resolved by monday(business hour)..

============================================show version==================

------------------ show version ------------------

Cisco Internetwork Operating System Software
IOS (tm) s72033_rp Software (s72033_rp-IPSERVICESK9-M), Version 12.2(18)SXF5, RELEASE SOFTWARE (fc3)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2006 by cisco Systems, Inc.
Compiled Sat 08-Jul-06 03:22 by kellythw
Image text-base: 0x40101040, data-base: 0x42D88000

ROM: System Bootstrap, Version 12.2(17r)SX5, RELEASE SOFTWARE (fc1)
BOOTLDR: s72033_rp Software (s72033_rp-IPSERVICESK9-M), Version 12.2(18)SXF5, RELEASE SOFTWARE (fc3)

NDAINFA6509-ODD uptime is 7 weeks, 9 hours, 17 minutes
Time since NDAINFA6509-ODD switched to active is 7 weeks, 9 hours, 16 minutes
System returned to ROM by reload at 06:23:20 UTC Tue Dec 22 2009 (SP by reload)
System restarted at 14:36:56 IST Fri Feb 12 2010
System image file is "sup-bootdisk:s72033-ipservicesk9-mz.122-18.SXF5.bin"

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

cisco WS-C6509-E (R7000) processor (revision 1.5) with 458720K/65536K bytes of memory.
Processor board ID SMG1324NM00
SR71000 CPU at 600Mhz, Implementation 0x504, Rev 1.2, 512KB L2 Cache
Last reset from s/w reset
SuperLAT software (copyright 1990 by Meridian Technology Corp).
X.25 software, Version 3.0.0.
Bridging software.
TN3270 Emulation software.
114 Virtual Ethernet/IEEE 802.3 interfaces
170 Gigabit Ethernet/IEEE 802.3 interfaces
1917K bytes of non-volatile configuration memory.
8192K bytes of packet buffer memory.

65536K bytes of Flash internal SIMM (Sector size 512K).
Configuration register is 0x2102

==================================show module=================

------------------ show module ------------------

Mod Ports Card Type                              Model              Serial No.
--- ----- -------------------------------------- ------------------ -----------
1   24 CEF720 24 port 1000mb SFP              WS-X6724-SFP       SAL09073Y9D
5    2 Supervisor Engine 720 (Active)         WS-SUP720-3B       SAL0937097Y
6   48 48-port 10/100/1000 RJ45 EtherModule   WS-X6148A-GE-TX    SAD103901B0
7   48 48-port 10/100/1000 RJ45 EtherModule   WS-X6148A-GE-TX    SAD09120DDU
8   48 48-port 10/100/1000 RJ45 EtherModule   WS-X6148A-GE-TX    SAL1027U8P4

Mod MAC addresses                       Hw    Fw           Sw           Status
--- ---------------------------------- ------ ------------ ------------ -------
1 0013.1a39.0bec to 0013.1a39.0c03   2.8   12.2(14r)S5 12.2(18)SXF5 Ok
5 0013.7f0d.8f1c to 0013.7f0d.8f1f   4.7   8.5(3)       12.2(18)SXF5 Ok
6 0019.2fa7.42c2 to 0019.2fa7.42f1   1.7   8.4(1)       8.5(0.46)RFW Ok
7 0013.8037.cff0 to 0013.8037.d01f   1.7   8.4(1)       8.5(0.46)RFW Ok
8 0018.7386.2970 to 0018.7386.299f   1.4   8.4(1)       8.5(0.46)RFW Ok

Mod Sub-Module                  Model              Serial       Hw     Status
---- --------------------------- ------------------ ----------- ------- -------
1 Centralized Forwarding Card WS-F6700-CFC       SAL09306BA5 2.1    Ok
5 Policy Feature Card 3       WS-F6K-PFC3B       SAL0936005L 2.5    Ok
5 MSFC3 Daughterboard         WS-SUP720          SAL093708ZY 2.9    Ok

Mod Online Diag Status
---- -------------------
1 Pass
5 Pass
6 Pass
7 Pass
8 Pass

===================================error logs======================

00:00:02: BaseBoard Index:156

00:00:02: DaughterBoard Index:208 (Centralized Forwarding Card)

00:00:02: Gemini Rev#: 3

Firmware compiled 23-Jun-06 11:59 by integ Build [100]

00:00:04: %SYS-CFC1-5-RESTART: System restarted --

Cisco Internetwork Operating System Software

IOS (tm) c6lc2 Software (c6lc2-SP-M), Version 12.2(18)SXF5, RELEASE

SOFTWARE (fc

3)

Technical Support: http://www.cisco.com/techsupport

Compiled Sat 08-Jul-06 03:22 by kellythw

*Nov 30 00:00:02.147: CFC1: Currently running ROMMON from S (Gold) region

Apr 2 07:05:42.536 UTC: %DIAG-SP-6-RUN_MINIMUM: Module 1: Running Minimal

Diagn

ostics...

013364: Apr 2 07:05:50.052 UTC: %MLS_RATE-4-DISABLING: The Layer2 Rate

Limiters

have been disabled.

013365: Apr 2 07:05:50.300 UTC: %LINK-3-UPDOWN: Interface

GigabitEthernet1/1, c

hanged state to down

013366: Apr 2 07:05:50.500 UTC: %LINK-3-UPDOWN: Interface

GigabitEthernet1/2, c

hanged state to down

Apr 2 07:05:49.680 UTC: %DIAG-SP-6-DIAG_OK: Module 1: Passed Online

Diagnostics

013367: Apr 2 07:05:51.104 UTC: %LINK-3-UPDOWN: Interface

GigabitEthernet1/3, c

hanged state to down

013368: Apr 2 07:05:51.308 UTC: %LINK-3-UPDOWN: Interface

GigabitEthernet1/4, c

hanged state to down

013369: Apr 2 07:05:51.512 UTC: %LINK-3-UPDOWN: Interface

GigabitEthernet1/5, c

hanged state to down

013370: Apr 2 07:05:51.716 UTC: %LINK-3-UPDOWN: Interface

GigabitEthernet1/6, c

hanged state to down

013371: Apr 2 07:05:51.920 UTC: %LINK-3-UPDOWN: Interface

GigabitEthernet1/7, c

hanged state to down

013372: Apr 2 07:05:52.124 UTC: %LINK-3-UPDOWN: Interface

GigabitEthernet1/8, c

hanged state to down

013373: Apr 2 07:05:52.328 UTC: %LINK-3-UPDOWN: Interface

GigabitEthernet1/9, c

hanged state to down

013374: Apr 2 07:05:52.504 UTC: %LINK-3-UPDOWN: Interface

GigabitEthernet1/4, c

hanged state to up

013375: Apr 2 07:05:52.508 UTC: %LINK-3-UPDOWN: Interface

GigabitEthernet1/5, c

hanged state to up

013376: Apr 2 07:05:52.532 UTC: %LINK-3-UPDOWN: Interface

GigabitEthernet1/10,

changed state to down

013377: Apr 2 07:05:52.736 UTC: %LINK-3-UPDOWN: Interface

GigabitEthernet1/11,

changed state to down

013378: Apr 2 07:05:52.940 UTC: %LINK-3-UPDOWN: Interface

GigabitEthernet1/12,

changed state to down

013379: Apr 2 07:05:53.024 UTC: %LINK-3-UPDOWN: Interface

GigabitEthernet1/7, c

hanged state to up

013380: Apr 2 07:05:53.144 UTC: %LINK-3-UPDOWN: Interface

GigabitEthernet1/13,

changed state to down

013381: Apr 2 07:05:53.348 UTC: %LINK-3-UPDOWN: Interface

GigabitEthernet1/14,

changed state to down

Apr 2 07:05:52.504 UTC: %LINK-SP-3-UPDOWN: Interface GigabitEthernet1/4,

change

d state to up

Apr 2 07:05:52.508 UTC: %LINK-SP-3-UPDOWN: Interface GigabitEthernet1/5,

change

d state to up

Apr 2 07:05:53.024 UTC: %LINK-SP-3-UPDOWN: Interface GigabitEthernet1/7,

change

d state to up

013382: Apr 2 07:05:53.552 UTC: %LINK-3-UPDOWN: Interface

GigabitEthernet1/15,

changed state to down

013383: Apr 2 07:05:53.756 UTC: %LINK-3-UPDOWN: Interface

GigabitEthernet1/16,

changed state to down

013384: Apr 2 07:05:53.960 UTC: %LINK-3-UPDOWN: Interface

GigabitEthernet1/17,

changed state to down

013385: Apr 2 07:05:54.076 UTC: %LINK-3-UPDOWN: Interface

GigabitEthernet1/11,

changed state to up

013386: Apr 2 07:05:54.076 UTC: %LINK-3-UPDOWN: Interface

GigabitEthernet1/13,

changed state to up

013387: Apr 2 07:05:54.164 UTC: %LINK-3-UPDOWN: Interface

GigabitEthernet1/18,

changed state to down

013388: Apr 2 07:05:54.368 UTC: %LINK-3-UPDOWN: Interface

GigabitEthernet1/19,

changed state to down

NDAINFA6509-ODD#

=======================================XXXXXXXXXXXXXXXXXXXXX==================================

Regards

Amar

Reza Sharifi · ‎04-02-2010

Hi Amar,

The logs really don't give us much info. One thing I noticed is that you are running an older firmware for the module and also an older IOS version. So you may want to start with upgrading the firmware for that module, if that does not solve the issue maybe upgrading the IOS will help. The latest IOS version on CCO is 12.2(33)SXI3.

HTH

Reza

amardram123 · ‎04-02-2010

Hi Reza,

do you have steps to upgrade the firmware..?

Regards

Amar

Giuseppe Larosa · ‎04-02-2010

Hello Amar,

try to have the linecard extracted and reseated

if behaviour does not change open an RMA

Hope to help

Giuseppe

amardram123 · ‎04-04-2010

Hi,

I extracted and reinserted, but then in 10 minutes module rebooted thrice.. then i shifted the line card from slot 1 to slot 2 and its working since last 2 days..:

But now i am getting error message .. pasted below:

024275: Apr 3 06:16:01.522 UTC: %SYS-5-CONFIG_I: Configured from console

by cis

co onvty0 (10.135.155.81)

Apr 3 09:24:20.862 UTC: %PM_SCP-SP-2-LCP_FW_ERR_INFORM: Module 2 is

experiencin

g the following error: RO[1] (162384 noncritical int in the last 10s, they

are n

ow disabled). ROINTMSK[1]:

2E9=C,F=728,24=1183,E8=4,52=0,4C=1E,49=0,9D=2183,9=0,

C=0.

Need help..

Regards

Amar

Ganesh Hariharan · ‎04-04-2010

Hi,
I
extracted and reinserted, but then in 10 minutes module rebooted
thrice.. then i shifted the line card from slot 1 to slot 2 and its
working since last 2 days..:
But now i am getting error message .. pasted below:
024275: Apr 3 06:16:01.522 UTC: %SYS-5-CONFIG_I: Configured from console
by cis
co onvty0 (10.135.155.81)
Apr 3 09:24:20.862 UTC: %PM_SCP-SP-2-LCP_FW_ERR_INFORM: Module 2 is
experiencin
g the following error: RO[1] (162384 noncritical int in the last 10s, they
are n
ow disabled). ROINTMSK[1]:
2E9=C,F=728,24=1183,E8=4,52=0,4C=1E,49=0,9D=2183,9=0,
C=0.
Need help..
Regards
Amar

Hi Amar,

See the below information from Cisco interpreter about the error messgage which is pasted above:-

%PM_SCP-6-LCP_FW_ERR_INFORM (x1): Module [dec] is experiencing the following error: [chars]
Explanation: This message indicates that the firmware of the module detected an error condition. The module is informing the supervisor engine about the error condition. [dec] is the module number, and [chars] is the error. This could be a transient issue.

Recommended Action:

1. Try resetting the module (soft-reset) using the command hw-module reset.

2. If the error message still displayed Power down the switch. Do the hard reset by pulling the module and reseating again. Power on the switch and monitor the error message.

3. If the error persists, try reseating the module in another free slot. If the error still displayed when moved to other module then there is no fault with the
chassis but module might be faulty.

4. Issue the command show test {module_number} before and after physically reseating the module to make sure module is not faulty. Make sure you have configured set test diaglevel complete command (a reset is required to enable this diagnostic mode).

5. Submit the show tech-support command output into the Output Interpreter to check for any errors.

6. If the error still persists with the module, you may have to replace the module.

Hope to Help !!

Ganesh.H

Cisco are currently donating money to the Haiti earthquake appeal for every rating so please consider rating all helpful posts.

madhusudhan s · ‎04-06-2010

Hi all,

After i moved the WS-X6724-SFP module in diffrent slot it worked good for 2 days and then same issue.module started resetting.. We have replaced the module and it is stable now..

regard

amar

Ganesh Hariharan · ‎04-06-2010

Hi all,

After i moved the WS-X6724-SFP module in diffrent slot it worked good for 2 days and then same issue.module started resetting.. We have replaced the module and it is stable now..

regard

amar

Hi Amar,

Then as per your observation most probably is the issue is with the module only,I would suggest just monitor for another two days and share your observation on the same with us.

Ganesh.H