Core Switch Migration

[prekojo]

We are trying starting a core switch migration from a 4507 to a 4510.  The two switches are links together by a vlan trunk that is also a port channel.  We also have a port channel running from the 4510 up to our UCS server environment.  All fiber uplinks for our user access switches are connected to the 4507.  Currently for our users to access all servers they hit the 4507, then cross the port channel, hit the 4510 and access all servers.  This works without any issues.  Our goal is to migrate all the user switch fiber uplinks from the 4507 over to the 4510 and remove the 4507.

We started this progress, but unplugging the first fiber uplink from the 4507 that connect to a user switch (3750), and plugging in into the 4510.  Less than a minute after plugging the fiber into the 4510, we start getting sporadic activity across all our vlans and some vlans we lose all connectivity.  If we try to reverse the change by plugging the fiber uplink back into the 4507, the network activity remains sporadic and connectivity to some vlans is still completely lost.  We have to bounce the port channel between the two switches for resume normal network activity.

Any ideas on why moving just one fiber uplink from a user access switch from 4507 to the 4510, would cause this outage.  Please see the attached diagram for environment details.

Thanks

[Jon Marshall]

Joe

To me it looks like 4507 (coacoresw1) is the root bridge for all vlans.

From the perspective of both the 4500s yes it does.

Would be interested to hear how it goes and if you do see the same issue how you fixed it.

Jon

[willwetherman]

The behaviour that you described sounds like the direct result of a broadcast storm which can be caused by a network loop but this is odd if you dont have a loop in your network.

Having to bounce the port-channel to clear the fault does sound odd unless the port-channel links were somehow contributing to the issue.

Can you post your port-channel 12 configuration including the logical port-channel and memeber interfaces. Can you also post the output of 'show etherchannel summary' from both core switches?

[prekojo]

Attached is the config of the port channel and its member interfaces of both core switch.  Thanks for the response.

Joe

[willwetherman]

The configuration looks ok. I would normally use LACP when configuring etherchannels to protect against incompatibilities and mis-configurations etc but I cant see any obvious errors with the config here that would be contributing to the observed behavior.

Does spanning tree on the core switches see the logical port-channel interface and not the member interfaces? Can you post an output of 'show spanning-tree vlan xxx' from each core for one of the VLANs?

[Jon Marshall]

Hi Will

Does spanning tree on the core switches see the logical port-channel interface and not the member interfaces? Can you post an output of 'show spanning-tree vlan xxx' from each core for one of the VLANs?

It does from the perspective of the new switch ie. see the STP summary outputs posted above.

Have to admit this has got me confused as i can't see any obvious loops anywhere (VMware setup aside as i can't say). I initially thought of wrong mac address tables but STP TCNs should sort that out and the problems seems to be with the servers more than anything rather than the users.

It does show all the symptoms of an STP loop but i can't see where one would be created by moving a switch which only has one connection anyway.

Jon

[prekojo]

Hi Will.  Attached is the output of the show spanning tree from two of the vlans on both switches.

Joe

[willwetherman]

Yeah it looks ok. I cant think what could be causing this.

As with Jon, I would be interested to hear how it goes when you test again and if you find the cause.

[prekojo]

Just another thought.  The 4507 is running VTP with pruning enabled.  The 3750 we are trying to move is VTP client for the 4507.  The 4510 is not running VTP and we will not be enabled VTP on the 4510.  Any change this would affect this issue?

Joe