Added switch to 3750 stack, now operating "half"?

Michael Hutt · ‎05-30-2010

Hey everybody,

I just added a new switch to our stack in our server room. The stack had 5 members, and the 6th was a member from another stack in another server room.

Here's what I did:

Powered off the switch, removed it from the stack
Deleted all config data off the switch leaving only the image
Set all the Switch [x] priority/provision/renumber commands to default
Connected the switch to the new stack
- I only did this with one Stackwise cable because I wasn't sure if I needed to connect both right away. It looked like this:
  - Switch 1 connected to Switch 2
  - Switch 2 connected to Switch 3
  - Switch 3 connected to Switch 4
  - Switch 4 connected to Switch 5
  - Switch 5 not connected to Switch 6
  - Switch 6 connected to Switch 1
Applied power to the new switch (Switch 6)

Everything booted properly and the stack reformed with the new 6th Switch, however the connection between Switches 4 and 5 is now in a "down" state, even after connecting the cable again from 5 to 6. Note: I did not originally do anything to the cable between Switches 4 and 5.

         Stack Port Status             Neighbors
Switch# Port 1     Port 2           Port 1   Port 2
--------------------------------------------------------
1        Ok         Ok                    2        6
2        Ok         Ok                    1        3
3        Ok         Ok                    4        2
4        Ok        Down                 3      None
5       Down      Ok                 None     6
6        Ok         Ok                    5        1

Did this happen because I booted the new switch with only one port connected to the stack?

What will happen if I reload switch 6? Won't switches 4 and 5 lose connectivity to one another because they don't have a link between themselves?

I can't really reload this entire stack because it's switching for our server room and those servers need to be up 24/7.

Ganesh Hariharan · ‎05-30-2010

Hey everybody,

I just added a new switch to our stack in our server room. The stack had 5 members, and the 6th was a member from another stack in another server room.

Here's what I did:

Powered off the switch, removed it from the stack
Deleted all config data off the switch leaving only the image
Set all the Switch [x] priority/provision/renumber commands to default
Connected the switch to the new stack
- I only did this with one Stackwise cable because I wasn't sure if I needed to connect both right away. It looked like this:
  - Switch 1 connected to Switch 2
  - Switch 2 connected to Switch 3
  - Switch 3 connected to Switch 4
  - Switch 4 connected to Switch 5
  - Switch 5 not connected to Switch 6
  - Switch 6 connected to Switch 1
Applied power to the new switch (Switch 6)

Everything booted properly and the stack reformed with the new 6th Switch, however the connection between Switches 4 and 5 is now in a "down" state, even after connecting the cable again from 5 to 6. Note: I did not originally do anything to the cable between Switches 4 and 5.

         Stack Port Status             Neighbors
Switch# Port 1     Port 2           Port 1   Port 2
--------------------------------------------------------
1        Ok         Ok                    2        6
2        Ok         Ok                    1        3
3        Ok         Ok                    4        2
4        Ok        Down                 3      None
5       Down      Ok                 None     6
6        Ok         Ok                    5        1

Did this happen because I booted the new switch with only one port connected to the stack?

What will happen if I reload switch 6? Won't switches 4 and 5 lose connectivity to one another because they don't have a link between themselves?

Hi,

Check out the below link for adding and removing a switch from the stack.

http://www.cisco.com/en/US/docs/switches/lan/catalyst3750/software/release/12.1_19_ea1/configuration/guide/swstack.html

Hope to help !!

Ganesh.H

Michael Hutt · ‎05-31-2010

Hi Ganeshh,

that's the guide I used to add the 6th switch to the stack. I didn't touch the connection between Switches 4 and 5. Compatibility-wise, everything seems OK: All 5 Switches have the same IOS image and they stack just fine:

                                           H/W   Current
Switch# Role   Mac Address     Priority Version State
----------------------------------------------------------
*1       Master 0012.d93b.5380     15     0       Ready
2       Member 0014.6a75.0580     14     0       Ready
3       Member 0015.f9bb.d980     13     0       Ready
4       Member 0015.f97b.7e00     12     0       Ready
5       Member 0018.1900.df80     11     0       Ready
6       Member 0017.e0de.1100     10     0       Ready

The problem is that the link between 4 and 5 went down during the process. Because Switch 5's only connection to the stack is through Switch 6, I cannot reasonably "undo" what I did, because doing so would mean all the servers plugged in to Switch 5 would lose connectivity, and we can't have that.

I've checked the cable between Switches 4 and 5, connected it and reconnected it a dozen times, making sure the screws were finger tight and uniform all around, with no wiggle on the connector. I've even exchanged the cable with a cable I know to work, but the link is still down.

I've thought about reloading Switch 6, but I can only assume that doing so would also cause Switch 5 to lose connectivity to the stack and cause lots of problems.

Edit: To be more accurate, I looked through the logs and found that the Stack Port connection between switches 4 and 5 went down shortly after Switch 6 came up and joined the stack. Like I said before, I waited to complete the ring until after Switch 6 had joined. Until that point, Switch 6 was connected to Switch 1. It wasnt until I connected Switch 6 to Switch 5 that the link between Switch 5 and Switch 4 went down.

Logs:

May 29 12:54:42: %STACKMGR-5-SWITCH_READY: Switch 6 is READY (RT-BACKBONE-6)
May 29 12:56:49: %STACKMGR-4-STACK_LINK_CHANGE: Stack Port 2 Switch 4 has changed to state DOWN
May 29 12:56:50: %STACKMGR-4-STACK_LINK_CHANGE: Stack Port 1 Switch 6 has changed to state UP
May 29 12:56:50: %STACKMGR-4-STACK_LINK_CHANGE: Stack Port 1 Switch 5 has changed to state DOWN
May 29 12:56:50: %STACKMGR-4-STACK_LINK_CHANGE: Stack Port 2 Switch 5 has changed to state UP

As you can see, Switch 6 joined the stack without problem, but as soon as I closed the ring, an unrelated link went down. When I plugged in Port 2 Switch 5 > Port 1 Switch 6, Port 2 Switch 4 > Port 1 Switch 5 went down. All screws are tight (not too tight, just checked all of them) and no cables are loose.

Michael Hutt · ‎06-08-2010

I still have this problem. I don't know why the link between 4 and 5 is down; I didn't even touch that link when adding the new switch. I'm afraid that if I attempt to remove or restart Switch 6 that Switch 5 will lose connectivity.

Does anyone have any recommendations?

Michael Hutt · ‎06-08-2010

If I ran the commands:

3750-Stack#diagnostic start switch 4 test 1

3750-Stack#diagnostic start switch 5 test 1

to run the TestPortAsicStackPortLoopback test, what would happen? They're classified as non-disruptive tests, but the whole addition of a new stack member was supposed to also be non-disruptive and now it ain't working properly, so I don't really want to believe the documentation at this point.

If I run this loopback test in order to check to see if the actual stack-ports are faulty, would it cause the switches to lose data or disrupt network traffic in any way?