Re: Cards in "other" status - Catalyst 6509

jefforsi · ‎07-09-2003

Hi,

I have a Catalyst 6509 with the following modules and software versions:

show mod

Mod Slot Ports Module-Type Model Sub Status

--- ---- ----- ------------------------- ------------------- --- --------

1 1 2 1000BaseX Supervisor WS-X6K-SUP1A-2GE yes ok

15 1 1 Multilayer Switch Feature WS-F6K-MSFC no other

2 2 2 1000BaseX Supervisor WS-X6K-SUP1A-2GE yes other

16 2 1 Multilayer Switch Feature WS-F6K-MSFC no other

3 3 48 10/100BaseTX Ethernet WS-X6248-RJ-45 no other

Mod Module-Name Serial-Num

--- -------------------- -----------

1 SAD04230B23

15 SAD04170BCX

2 SAD04130EH1

16 SAD04130DGA

3 SAD0349075K

Mod MAC-Address(es) Hw Fw Sw

--- -------------------------------------- ------ ---------- -----------------

1 00-d0-c0-cf-3b-ca to 00-d0-c0-cf-3b-cb 3.1 5.3(1) 6.3(7)

00-d0-c0-cf-3b-c8 to 00-d0-c0-cf-3b-c9

00-d0-04-58-a0-00 to 00-d0-04-58-a3-ff

15 00-d0-d3-38-b4-e8 to 00-d0-d3-38-b5-27 1.4 12.1(11b)E 12.1(11b)E4

2 00-30-96-32-71-62 to 00-30-96-32-71-63 3.1 5.3(1) 6.3(7)

00-30-96-32-71-60 to 00-30-96-32-71-61

16 00-30-96-32-71-64 to 00-30-96-32-71-a3 1.4 12.1(11b)E 12.1(11b)E4

3 00-d0-d3-9f-9d-0c to 00-d0-d3-9f-9d-3b 1.1 4.2(0.24)V 6.3(7)

Mod Sub-Type Sub-Model Sub-Serial Sub-Hw

--- ----------------------- ------------------- ----------- ------

1 L3 Switching Engine WS-F6K-PFC SAD0423095L 1.1

2 L3 Switching Engine WS-F6K-PFC SAD041303AV 1.1

Console> (enable) show ver

WS-C6509 Software, Version NmpSW: 6.3(7)

NMP S/W compiled on May 16 2002, 17:32:59

System Bootstrap Version: 5.3(1)

Hardware Version: 2.0 Model: WS-C6509 Serial #: SCA050600J4

PS1 Module: WS-CAC-1300W Serial #: ACP04480017

Mod Port Model Serial # Versions

--- ---- ------------------- ----------- --------------------------------------

1 2 WS-X6K-SUP1A-2GE SAD04230B23 Hw : 3.1

Fw : 5.3(1)

Fw1: 5.1(1)CSX

Sw : 6.3(7)

Sw1: 6.3(7)

WS-F6K-PFC SAD0423095L Hw : 1.1

2 2 WS-X6K-SUP1A-2GE SAD04130EH1 Hw : 3.1

Fw : 5.3(1)

Fw1: 5.1(1)CSX

Sw : 6.3(7)

Sw1: 6.3(7)

WS-F6K-PFC SAD041303AV Hw : 1.1

3 48 WS-X6248-RJ-45 SAD0349075K Hw : 1.1

Fw : 4.2(0.24)VAI78

Sw : 6.3(7)

15 1 WS-F6K-MSFC SAD04170BCX Hw : 1.4

Fw : 12.1(11b)E4

Sw : 12.1(11b)E4

16 1 WS-F6K-MSFC SAD04130DGA Hw : 1.4

Fw : 12.1(11b)E4

Sw : 12.1(11b)E4

DRAM FLASH NVRAM

Module Total Used Free Total Used Free Total Used Free

------ ------- ------- ------- ------- ------- ------- ----- ----- -----

1 65408K 47338K 18070K 16384K 9476K 6908K 512K 238K 274K

Uptime is 0 day, 20 hours, 0 minute

-----------------------------------------------------------------------------------------------------

How you can see, at the output from "show module" command, some cards appear as "other" state... I left this switch running various "ping's" between his cards to test his modules last night...

look at the "show logging buffer -1023" command:

Console> (enable) show logging buffer -1023

2003 Jul 08 20:25:26 %SYS-1-SYS_ENABLEPS: Power supply 1 enabled

2003 Jul 08 20:25:27 %SYS-5-SUP_NONHASWOVER:Supervisor becoming active(non-HA switchover)

2003 Jul 08 20:25:48 %MLS-5-MCAST_STATUS:IP Multicast Multilayer Switching is enabled

2003 Jul 08 20:25:48 %MLS-5-NDEDISABLED:Netflow Data Export disabled

2003 Jul 08 20:25:49 %SYS-5-MOD_OK:Module 1 is online

2003 Jul 08 20:25:49 %SYS-3-MOD_PORTINTFINSYNC:Port Interface in sync for Module 1

2003 Jul 08 20:25:50 %SYS-5-MOD_OK:Module 3 is online

2003 Jul 08 20:25:52 %SYS-5-MOD_OK:Module 15 is online

2003 Jul 08 20:27:06 %SYS-5-MOD_OK:Module 15 is online

2003 Jul 08 20:27:36 %SYS-5-SUP_MODSBY:Module 2 is in standby mode

2003 Jul 08 20:27:46 %SYS-5-SUP_IMGSYNC:File synchronization process will start in 10 seconds

2003 Jul 08 20:27:53 %SYS-5-PORT_SSUPOK:Ports on standby supervisor (module 2) are up

2003 Jul 08 20:27:53 %SYS-3-MOD_PORTINTFINSYNC:Port Interface in sync for Module 2

2003 Jul 08 20:28:15 %SYS-5-SUP_IMGSYNCSTART:Active supervisor is synchronizing the NMP image

2003 Jul 08 20:28:29 %SYS-5-MOD_OK:Module 16 is online

2003 Jul 08 20:28:31 %SYS-5-SUP_IMGSYNCFINISH:Active supervisor has synchronized the NMP image

2003 Jul 08 20:28:46 %SYS-5-MOD_OK:Module 15 is online

2003 Jul 08 20:31:26 %SYS-5-MOD_RESET:Module 15 reset from Console//

2003 Jul 08 20:32:07 %SYS-5-MOD_OK:Module 15 is online

2003 Jul 08 20:32:19 %SYS-5-MOD_RESET:Module 15 reset from Console//

2003 Jul 08 20:33:17 %SYS-3-MOD_FAIL:Module 15 failed to come online

2003 Jul 08 20:33:23 %SYS-5-MOD_RESET:Module 15 reset from Console//

2003 Jul 08 20:34:23 %SYS-5-MOD_OK:Module 15 is online

--More-- 2003 Jul 08 20:36:03 %SYS-5-MOD_OK:Module 15 is online

2003 Jul 08 20:36:56 %SYS-5-MOD_OK:Module 15 is online

2003 Jul 08 21:17:33 %SYS-5-MOD_OK:Module 16 is online

2003 Jul 08 21:18:52 %SYS-5-MOD_OK:Module 16 is online

2003 Jul 08 21:28:38 %PAGP-5-PORTTOSTP:Port 3/1 joined bridge port 3/1

2003 Jul 08 21:36:38 %SYS-5-MOD_OK:Module 15 is online

2003 Jul 09 03:02:12 %SYS-5-MOD_NOSCPPINGRESPONSE:Module 3 not responding... resetting module

2003 Jul 09 03:02:12 %SYS-5-MOD_RESET:Module 3 reset from Software

2003 Jul 09 03:02:14 %SYS-5-MOD_NOSCPPINGRESPONSE:Module 16 not responding... resetting module

2003 Jul 09 03:02:14 %SYS-5-MOD_RESET:Module 16 reset from Software

2003 Jul 09 03:02:41 %PAGP-5-PORTFROMSTP:Port 3/1 left bridge port 3/1

2003 Jul 09 03:03:07 %SYS-2-MOD_NOSCPRESPONSE:Module 2 not responding over SCP

2003 Jul 09 03:05:07 %SYS-2-MOD_NOSCPRESPONSE:Module 2 not responding over SCP

This error on Module 2 cannot communicate via SCP appear many times... I read on Error Decode Message on TAC that this error occur for example when the switch's bus is oversubscribed, ...

So I try to reset the sup in slot 2 by pressing the reset button on the front panel, but apparently the card cannot come up... all LED's are in ORANGE except the STATUS LED (GREEN)...

What's is the problem??? And how resolve it???

Prashanth Krishnappa · ‎07-09-2003

I have seen similar issues due to the following bug

http://www.cisco.com/cgi-bin/bugtool/onebug.pl?bugid=CSCea38268

Pull out one card at a time and see if the problem goes away

The code fix which is not yet available will intelligently detect module which could be causing thebus to go bad and shut it down.

jefforsi · ‎07-09-2003

OK,

but the logging message that I have is different from the message displayed in the BUG... the problem is similar... what's is the "More" status in BUG ToolKit???

I try to reset the system, but now... the mod 3 (48 RJ-45 ports) still in "other" state...

the standby supervisor (slot 2) cannot come up...

I will try to reset the system with only slot 1 plugged.. and I'll insert the other modules later...

Do you have any other idea?

Prashanth Krishnappa · ‎07-09-2003

When a bug is fixed and tested in some but not all versions effected, the assigned engineer may choose to place the report in the "More" state so the fixed code can be integrated into the appropriate releases. He or she will move the report to the "Resolved" state when all versions are fixed.

This is more of one bad card causing the entire bus to hang. Try booting switch with one card at a time(you have only 3 and should be fairly easy)

jefforsi · ‎07-09-2003

I try to reset the system with only 1 supervisor in slot 1... the system starts ok, but the MSFC on this supervisor cannot come up... in the show module command:

Mod Slot Ports Module-Type Model Sub Status

--- ---- ----- ------------------------- ------------------- --- --------

1 1 2 1000BaseX Supervisor WS-X6K-SUP1A-2GE yes ok

Mod Module-Name Serial-Num

--- -------------------- -----------

1 SAD04230B23

Mod MAC-Address(es) Hw Fw Sw

--- -------------------------------------- ------ ---------- -----------------

1 00-d0-c0-cf-3b-ca to 00-d0-c0-cf-3b-cb 3.1 5.3(1) 6.3(7)

00-d0-c0-cf-3b-c8 to 00-d0-c0-cf-3b-c9

00-d0-04-58-a0-00 to 00-d0-04-58-a3-ff

Mod Sub-Type Sub-Model Sub-Serial Sub-Hw

--- ----------------------- ------------------- ----------- ------

1 L3 Switching Engine WS-F6K-PFC SAD0423095L 1.1

the MSFC does not appear...

Prashanth Krishnappa · ‎07-09-2003

Try "switch console" from a console session

http://www.cisco.com/warp/public/473/124.html

jefforsi · ‎07-09-2003

Both switch console and session 15 does not help....

jefforsi · ‎07-09-2003

I removed the supervisor from slot 1... the reset the system with only the supervisor on the slot 2 (this supervisor is in "other" state before), the system started ok... the I inserted the slot 3 (48 RJ45), the mod inserted ok... now I inserted the supervisor in slot 1 and come up ok too....

What's is the mistery??? It's a MAGIC??

Prashanth Krishnappa · ‎07-09-2003

Monitor it for a while(few hours). It could be due to a poorly seated card as well. Make sure the cards are seated fully with the thumb screws tight.

jefforsi · ‎07-09-2003

ok....

This error is very strange...

thanks for while...