Thanks for your action, Andrew!

You are completely right, MSTP mode used.

When it works, show spanning-tree information looks like in attached file (output is large enought, to harm this post).

But when it turns off (due to it's own initiative):

10.255.0.41(config)# do sh spanning-tree active

Spanning tree disabled (BPDU filtering) mode MSTP
Default port cost method:  long

10.255.0.41(config)#

There are no multiple links between switches, i tried to draw basic scheeme, it is in attached picture.

1. Every switch in "Access Layer" is sps224g4.

2. Every such ring (the only difference between picture and real network - that there are 8-12 switches in one garland on access switches) - is separate L2 segment.

3. NO customer switches appear.

4. NO additional non-mstp capable devices used.

5. If failure happens, it affects aggregation swithes pair as well

Only two physical interfaces are loaded up to 1Gbps (one to access layer switches, and second - to neighboring aggregation switch), but telnet/ssh unavailable.

In fact, there is an opinion, that these two affected in a way of ping efforts from outer monitoring system. Aggregation switches are the default gateways for access layer switches (VRRP/HSRP mechanism), and when the icmp packet comes from outside - arp request are produced in layer2 segment. As I can to assume - 1Gbps of arp requests in looped network have to be processed by 400MHz aggregation switches CPUs, which cannot handle such intensive arp flow, and fall down.

6. G3 and G4 directed to two different aggregation switches.

7. Yes, spanning tree root bridge reachable and displayed in "sh spanning-tree active" output (attached) correctly.

8. Yes, root bridge is static - one of aggregation switches pair configured with priority of 4096. Failover configuration applied on second aggregation, with the bridge priority of 8192. CIST root is a part of CORE layer.
9. No logs related to stp even by somehow, the only messages that appeared in log when failure happens are related to GVRP.

GVRP actions performed because of when stp goes to disabled state and bpdu's blocked, neighboring switches cannot send bpdu's one to another, and assume that there is no rings anymore. Due to this, some ports in entire garland, blocked for different MSTP Instances, become to forwarding state and from that moment GVRP messages can travel over the network in any direction.

At the same moment loop occurs.

Thanks for the updates.

Yep, nothing is overly obvious from this info.  We might need to open a case and inquire of any known bugs related to MSTP ... I do not know of any.  If the spanning tree process is encountering a problem, it should not become disabled.  This would be a bug ... but again, I do not know.

Odd that one of the ports is not listed as alternate ... if both ports have paths to the root, only one should be forwarding and the other should be ALT.  It seems like the one port is designated, meaning that other switches connect through this one to get to the core.  Not sure you want these edge switches to be transit switches ...

Interesting thought on the ping, not sure ...an arp broadcast should go out all ports, but not the one it was received on.  This could be a temporary problem ... but not likely a permanent one. What you are describing does sound like a loop though ...

When the problem occurs, do you have the logs from the agg switches?  Can you see which ports are causing the "overflow" (for lack of better terms ...)?

Forgive me for not fully understanding the diagram ... I have a question related to it.

In the diagram, is it true that each outer ring is a logical L2 network, but a physical multi-home edge?  If I understand you correctly, G3 and G4 links go upstream to the aggregation switches and not to other edge switches.  Is this correct? 

If there is a link to the other edge switches (as the diagram suggests), then I might suggest that a root failover could be the culprit. When a failover occurs, it seems that the traffic might traverse multiple times and in diff directions.  Each switch is confused as to which ports are forwarding.  The switch might shut down any process that is stalled ... (MSTP) and then this disables spanning tree.  Just a theory ..., not sure how close it is ...

Do you have to reboot the switch to regain MSTP, or does it come back on it's own?

Any chance you can test this?  Probably not in a production network but just wondering.  It would be good to have several mcast streams, and then change the root. You could also try a ping from an unknown source ....

Have you opened a case yet with our small biz support center?

What kind of switches do you have in the agg?  If Cisco, have you opened one with Cisco TAC as well?  I am not sure, however ... I am wondering if there is some incoming rate limiting or control-plane policing you should consider to protect your agg.  Just a thought.

I look forward to your next update.  Kindest regards

I have come back to this and re-read what you wrote and what I wrote.

When the switches no longer have spanning tree functioning correctly, then all port interfaces will be in forwarding state.  So yes, there are loops during this time; loops in the 'garland'.  Due to your design, even the agg switches will not know that they have a loop.

I think you should open a case so we can engage additional resources, and do please provide me the case number so I can also inquire and assist.

Have a great night,

Andrew

1. Agree, arp requests are forwarded to all ports (where corresponding vlan permitted), except one, where request originated from. But this request travels through entire 'garland' toward the aggregation switches, and enters the same garland from the another side, and comes to the same switch again and again. It is just a one version where is the source of 1Gbps traffic of ~850-1000 Mpps (measured from console RS232 port).

2. This switch not the one, who breaks the loop in stp ring, when stp on all devices operates correctly. On sps224g4, which blocks one of ports (the next device in 'garland'), g4 is a root port (state FRW) and g3 is indicated as alternate (state BLK). And on all other members of stp domain output of "sh spanning-tree active" looks definitely like the one, attached before.

3. There is no logs on aggregation switches related to spanning tree (even if stp debugging is on), but i gathered information about "overflow" ports manually from rs232 console (failure happens at about 2:00AM, so i had some time to inspect situation).

This "overflow" ports just indicating a lot of BROADCAST traffic (stp, as far as i know, seems to be a multicast, so this 1Gbps is not bpdu's flow). And, one of ports (on aggregation switch) reported about "error draining packets", when i tried to "shutdown" it. This message related on Broadcom's chip that handles physical interface, and not related to specific switch vendor. That was a REAL overflow. Actually, only two ports were 100% loaded at that time, and these two are:

a) one, who "looks" in a direction of sps224g4("left" ending of 'garland'), where stp failed.

b) second, who "looks" to second aggregation switch.

On second aggregation situation was similar? 100% loaded port are:

a) one, who looks in a direction of sps224g4("right" side of 'garland'), where stp failed.

b) second, who looks to fist aggregation switch.

4. About diagram:

If You mean that "edge" switches are the ones, where end-customers connected to, than scheeme is the following:

Aggregation switch 1 (port 1) <-> (g3) sps224g4 no.1 (g4) <-> (g3) sps224g4 no.2 (g4) <-> ...<-> (g3) sps224g4 no.10 (g4) <-> (port 1) Aggregation switch 2 (port 22). <-> (port 22) Aggregation switch 1

The part "(g3) sps224g4 no.1 (g4) <-> (g3) sps224g4 no.2 (g4) <-> ... <-> (g3) sps224g4 no.10 (g4)" - basically named as 'garland'.

5. I thought a lot about, who are the culprit... maybe a root switch, everything is possible, but, i have no idea, what kind of failure on "switch A" can be a reason for software error (such as disabling stp) on "switch B". The more so, when this two are divided by at least three another switches.

6. No there is no need to reboot switch to normalize it's operating mode. When loop breaked physically, intensive broadcast data flow dissapeared (as it has to be), and telnet/ssh becomes available. I can connect to sps224g4, just write "spanning-tree" in config mode and then press .

But, reboot also can be preferred in a way, that saved "startup-config" is NOT modified during failure, and during device's startup, configuration will we readed from flash, and succesfully applied with all directives needed (including stp).

First variant is going to be much faster.

This procedure is necessary. Switch does not come back to normal mode by its own.

7. Yes, there is a possibility to test any scheeme not in production network. I have 2 sps224g4's and one 3com 5500G-EI-SFP(like an aggreagation swtich) in test stand (and, except them, three kinds of another switches in quantity of 7). Totally, I have 10 switches (all are smart, mstp capable, and all can perform any operation, that sps224g4 can, except dvmrp), and I can to assebly any kind of constructor from them. Following to Your question - yes, there IS a possibility to perform some tests.

8. How case can be opened (i'll do that at the morning)? Over the phone only?

Maybe there is some emails exists?

Describing of problem can be more exact, if using email, some config files, command outputs and scheemes can be attached.. etc.

Any way, it should NOT be fun,  when i will not understand some word in english... when talking ober the phone..

Many thanks for the response.

Yes, 2AM sounds like a maintenance window, so your idea might be right and management traffic. When the switches think they are the root, or when they do not have spanning tree running, then they will forward out all interfaces ...This will create the storm.

The number of packets you mention makes me also wonder if the mcast streams are 'bleeding' over and being sent as well ... This is a lot of traffic when the problem hits.  Does your management station also use mcast to test?

Internally, I have escalated this to our tech support.  I hope they will join in shortly.

Concerning the testing, I was going to ask if we could force the STP to be turned off by confusing it with no bpdus or higher then lower priority bpdus ... , however after thinking about it some, I think we should hold off.

Lets wait till the support folk get involved.  They may request some testing, but I think it is best if they add in and make a suggestion.  Your environment is not the easiest to recreate, and you have a very extensive setup.  I think we should wait to hear from them.

Also, I want to thank you for all the clarification and information you have provided thus far.  Kindest regards and I hope to hear from you soon,

Andrew Lee Lissitz

Hello and Good afternoon,

I have spoken to our tech support, and they have asked for you to call the support number.  Please use this link to find the correct number for your country:

http://www.cisco.com/en/US/support/tsd_cisco_small_business_support_center_contacts.html

Can you also please send me a note with your case number and progress?  Do please let me know how I can help.  Kindest regards,

Andrew

Yes, ok, I will post case number here, when it will be created. Thank's once more!

Following the information, that had been received last week, Case ID of "Cisco 77" suggested. Also got some quantity of questions, now gathering answers.

Thanks for the update.  Good to hear that things are progressing along the right path.

No, management stations are not using mcast, just traditional operations like "icmp ping" or "snmp read" or "udp-traps receiving" performed.

Great thanks for all of actions, Andrew. It should be fine, if some researches will be committed.

From my side - any experiments/tests/actions can be performed, including firmware upgrades/downgrades, with or without multicast flows (there is a possibility to retransmit some quantity of real video streams) with all QoS rules etc... Actually, we can assembly a fully featured isolated little "copy" of enterprise network.

Best regards!