We just experienced a problem and caused production network down.
Here is the scenario:
Two 3750G/48 have stacked together, configured port channel include both port 49 (fiber), then connected to core router 6509 port 2/19 and 2/20, both ports are also in port channel. Running fine, about 100 machines are connected to this stacked switches.
One new 3750G/48 switch powered on and connected to the port 2/14 at the same router. This new switch has not been stacked to other two switches---means it is a standalone switch without any machine connected to. The router port was not turned on, but switch port 49 is on.
After turned on the router port 2/14 and configured as part of port channel with port 2/19 and 20, whole network (that core router supports) down immediately. Pull the fiber out from switch immediately, disconnected it from the network, the traffic come back.
From existing stacked switch, we saw many mac flapping info in the log:
Jun 17 14:58:25.933: %SW_MATM-4-MACFLAP_NOTIF: Host 0024.e834.9723 in vlan 110 is flapping between port Po1(note: this is switch's ether channel) and port Gi1/0/27 (note: this is port connected to the machine)
Jun 17 14:58:27.024: %SW_MATM-4-MACFLAP_NOTIF: Host 0024.e841.ad31 in vlan 111 is flapping between port Gi2/0/11 and port Po1
In the other switch, I saw:
Jun 17 14:58:53.830: %SW_MATM-4-MACFLAP_NOTIF: Host 0015.2b68.bf80 in vlan 113 is flapping between port Po2(note: this is to standby router 6509) and port Po1 (note: this is to active router 6509).
Jun 17 14:58:53.830: %SW_MATM-4-MACFLAP_NOTIF: Host 0018.8b31.e7cc in vlan 49 is flapping between port Po2 and port Po1
Jun 17 14:58:53.830: %SW_MATM-4-MACFLAP_NOTIF: Host 0001.d740.4c85 in vlan 921 is flapping between port Po1 and port Po2
Note(again):Here Po1 (etherport) is to active core router and Po2(etherport) is to standby core router.
Looks like in the procedure, a certain loop was formed.
Here are my Questions:
a. What was the problem to cause core router jammed and cannot handle traffic?
b. What is the correct procedure to add a new 3750G to existing stack?
c. Is the stackWise Cable hot swappable? What issues that we need to pay attention to add a new 3750G to the existing stack? Some coworkers suggested to config ether channel first, then connect physical fiber. Is it correct?
d. Last question: if we have 3 stacked 3750G, but for some reason, such as, stackwise cable broken, one switch is disconnected from the other switch in the stack, can this scenirio cause the network jam as I experienced?
Thank you very much for your time and help.
6509 IOS ver: 12.2(33)SXH5
3750G IOS ver: 12.2.50-SE3