C3750E-48PD stack unstable

filiberto.aguirre · ‎03-25-2012

Hi,

End user has two new 3750E switches. At the begining were working fine but suddenly loggs began showing the following messages

ar 24 17:32:02: %STACKMGR-4-STACK_LINK_CHANGE: Stack Port 1 Switch 1 has changed to state DOWN
Mar 24 17:32:03: %STACKMGR-4-STACK_LINK_CHANGE: Stack Port 1 Switch 1 has changed to state UP
Mar 24 17:32:33: %STACKMGR-4-STACK_LINK_CHANGE: Stack Port 2 Switch 2 has changed to state DOWN
Mar 24 17:32:34: %STACKMGR-4-STACK_LINK_CHANGE: Stack Port 2 Switch 2 has changed to state UP
Mar 24 17:35:15: %STACKMGR-4-STACK_LINK_CHANGE: Stack Port 2 Switch 2 has changed to state DOWN
Mar 24 17:35:16: %STACKMGR-4-STACK_LINK_CHANGE: Stack Port 2 Switch 2 has changed to state UP
Mar 24 18:01:35: %STACKMGR-4-STACK_LINK_CHANGE: Stack Port 2 Switch 2 has changed to state DOWN
Mar 24 18:01:36: %STACKMGR-4-STACK_LINK_CHANGE: Stack Port 2 Switch 2 has changed to state UP
Mar 24 17:32:02: %STACKMGR-4-STACK_LINK_CHANGE: Stack Port 1 Switch 1 has changed to state DOWN
Mar 24 17:32:03: %STACKMGR-4-STACK_LINK_CHANGE: Stack Port 1 Switch 1 has changed to state UP
Mar 24 17:32:33: %STACKMGR-4-STACK_LINK_CHANGE: Stack Port 2 Switch 2 has changed to state DOWN
Mar 24 17:32:34: %STACKMGR-4-STACK_LINK_CHANGE: Stack Port 2 Switch 2 has changed to state UP
Mar 24 17:35:15: %STACKMGR-4-STACK_LINK_CHANGE: Stack Port 2 Switch 2 has changed to state DOWN
Mar 24 17:35:16: %STACKMGR-4-STACK_LINK_CHANGE: Stack Port 2 Switch 2 has changed to state UP
Mar 24 18:01:35: %STACKMGR-4-STACK_LINK_CHANGE: Stack Port 2 Switch 2 has changed to state DOWN
Mar 24 18:01:36: %STACKMGR-4-STACK_LINK_CHANGE: Stack Port 2 Switch 2 has changed to state UP

I have powered down both switches and checked stack cables connection, but the problem remains.

Switch Ports Model SW Version SW Image

------ ----- ----- ---------- ----------

* 1 54 WS-C3750E-48PD 12.2(52)SE C3750E-UNIVERSALK9-M

2 54 WS-C3750E-48PD 12.2(52)SE C3750E-UNIVERSALK9-M

What else can I checked in order to clear this issue?

regards

Leo Laohoo · ‎03-25-2012

Make sure the stack cables are secure.

Judging from the error messages, looks like Switch #2 is faulty.

Tirumala Earla · ‎03-26-2012

With help of # show switch detail command you can see the status of the stack cable connetivity. Please verify and put output for reference.

filiberto.aguirre · ‎03-26-2012

Hi Tirumala,

This is the output I got some minuetes ago,

COREP5#      sh switch detail
Switch/Stack Mac Address : 0c85.25c3.9400
                                           H/W   Current
Switch# Role   Mac Address     Priority Version State
----------------------------------------------------------
*1       Master 0c85.25c3.9400     10     1       Ready
2       Member 0c85.25c2.b480     1      1       Ready

         Stack Port Status             Neighbors
Switch# Port 1     Port 2           Port 1   Port 2
--------------------------------------------------------
1        Ok         Ok                2        2
2        Ok         Ok                1        1

COREP5#      sh switch detail
Switch/Stack Mac Address : 0c85.25c3.9400
                                           H/W   Current
Switch# Role   Mac Address     Priority Version State
----------------------------------------------------------
*1       Master 0c85.25c3.9400     10     1       Ready
2       Member 0c85.25c2.b480     1      1       Ready

         Stack Port Status             Neighbors
Switch# Port 1     Port 2           Port 1   Port 2
--------------------------------------------------------
1        Ok         Ok                2        2
2        Ok         Ok                1        1

COREP5#      sh switch detail
Switch/Stack Mac Address : 0c85.25c3.9400
                                           H/W   Current
Switch# Role   Mac Address     Priority Version State
----------------------------------------------------------
*1       Master 0c85.25c3.9400     10     1       Ready
2       Member 0c85.25c2.b480     1      1       Ready

         Stack Port Status             Neighbors
Switch# Port 1     Port 2           Port 1   Port 2
--------------------------------------------------------
1        Ok         Ok                2        2
2        Ok         Ok                1        1

**************************************************************************************

COREP5#      sh switch detail
Switch/Stack Mac Address : 0c85.25c3.9400
                                           H/W   Current
Switch# Role   Mac Address     Priority Version State
----------------------------------------------------------
*1       Master 0c85.25c3.9400     10     1       Ready
2       Member 0c85.25c2.b480     1      1       Ready

         Stack Port Status             Neighbors
Switch# Port 1     Port 2           Port 1   Port 2
--------------------------------------------------------
1        Ok         Ok                2        2
2        Ok        Down             None     None    ------------>>> sometimes DOWN

********************************************************

COREP5#      sh switch detail
Switch/Stack Mac Address : 0c85.25c3.9400
                                           H/W   Current
Switch# Role   Mac Address     Priority Version State
----------------------------------------------------------
*1       Master 0c85.25c3.9400     10     1       Ready
2       Member 0c85.25c2.b480     1      1       Ready

         Stack Port Status             Neighbors
Switch# Port 1     Port 2           Port 1   Port 2
--------------------------------------------------------
1        Ok         Ok                2      None --------------------------------------->>> ????
2        Ok         Ok                1        1

COREP5#

**************************************************************

COREP5#      sh switch detail
Switch/Stack Mac Address : 0c85.25c3.9400
                                           H/W   Current
Switch# Role   Mac Address     Priority Version State
----------------------------------------------------------
*1       Master 0c85.25c3.9400     10     1       Ready
2       Member 0c85.25c2.b480     1      1       Ready

         Stack Port Status             Neighbors
Switch# Port 1     Port 2           Port 1   Port 2
--------------------------------------------------------
1        Ok         Ok                2        2
2        Ok         Ok              None     None   ---------->>>>>> ??????

COREP5#      sh switch detail
Switch/Stack Mac Address : 0c85.25c3.9400
                                           H/W   Current
Switch# Role   Mac Address     Priority Version State
----------------------------------------------------------
*1       Master 0c85.25c3.9400     10     1       Ready
2       Member 0c85.25c2.b480     1      1       Ready

         Stack Port Status             Neighbors
Switch# Port 1     Port 2           Port 1   Port 2
--------------------------------------------------------
1        Ok         Ok                2      None   ---------->>>????
2        Ok        Down               1      None --->>>>>>???

COREP5#

Leo Laohoo · ‎03-26-2012

Judging from this latest output, my initial recommendations are still the same:

1. Have you check to ensure all four cable sockets are firmly secure to each stack port?

2. Stack member #2 is failing. You could try a disruptive test if wish. Command is: diagnostic start switch 2 test all

WARNING: This is a DISRUPTIVE test.

Post the output to the command "sh diagn result switch 2".

filiberto.aguirre · ‎03-26-2012

Hi Leo,

Thanks a lot for your interest in this issue.

I have already checked several times all cable sockets. I powered off both switches and disconnected cables, then connect once again firmly secure.

Because of is a DISRUPTIVE test I need to try in no working hours. :-(

regards

Leo Laohoo · ‎03-26-2012

Thanks. I await for your response and result.

Leo Laohoo · ‎03-27-2012

May I also ask you to post the output to the command "sh post" please?

filiberto.aguirre · ‎03-27-2012

Hi leo,

I send you the sh post info:

I'm waiting the maintenance windows in order to get the info you asked me.

I really appreciate your time and help.

COREP5#sh post
Stored system POST messages:

Switch 1
---------

POST: CPU MIC register Tests : Begin
POST: CPU MIC register Tests : End, Status Passed

POST: MA BIST : Begin
POST: MA BIST : End, Status Passed

POST: TCAM BIST : Begin
POST: TCAM BIST : End, Status Passed

POST: SF ASIC BIST : Begin
POST: SF ASIC BIST : End, Status Passed

POST: Switch Fabric Memory Tests : Begin
POST: Switch Fabric Memory Tests : End, Status Passed

POST: CPU MIC interface Loopback Tests : Begin
POST: CPU MIC interface Loopback Tests : End, Status Passed

POST: PortASIC RingLoopback Tests : Begin
POST: PortASIC RingLoopback Tests : End, Status Passed

POST: Inline Power Controller Tests : Begin
POST: Inline Power Controller Tests : End, Status Passed

POST: Thermal, Fan Tests : Begin
POST: Thermal, Fan Tests : End, Status Passed

POST: PortASIC Port Loopback Tests : Begin
POST: PortASIC Port Loopback Tests : End, Status Passed

POST: EMAC Loopback Tests : Begin
POST: EMAC Loopback Tests : End, Status Passed

Switch 2
---------