strange but very important question with MST

Answered Question
Jun 1st, 2007

Hi,

our network consist of 2 switches 6513 SUP720 12.2(18) in distribution layer and a number of bladeservers with cisco switches. CIGESM 12.1(22)EA9.

every blade switch is connected to one 6513 with one etherchannel port (2 interfaces), one usual interface(dedicated for NFS) and additional one usuall interface (oracle cluster interconnection).

We're running MST spanning tree...

config is here:

Instance Vlans mapped

0 none

1 1-497,499-599,601-4094

2 600

3 498

6513 is root for all MST instances

Today a cable was reconnected to the interface on bladeswitch, that was not correctly configured with Oracle VLAN and after that ALL bladeswitches put their etherchannel connection and Oracle connection in err-disable mode: "loopback detected"

First important question:

Is it correct that I move all VLAN's, including VLAN1 from MST0? all switchblades are connected as "Pre-STD"

the MST config of blade switches is in attach.

I have this problem too.
0 votes
Correct Answer by Francois Tallet about 9 years 7 months ago

Yes.

If a bridge learns a mac address M on a port P (by receiving a frame with M as a source mac address), it means that in the current topology, the device associated with M is off port P. As a result, if the bridge later receives a frame with M as a destination address on port P, it's going to discard it because it knows that the frame will reach its destination without the need to be switched through the bridge.

The loopback message plays a little trick with the learning function of the bridge as its source mac address is the same as its destination mac address. If the cam table does not contain the mac address of the loopback message, it is flooded. After that, it's going to be filtered. So the chance of a loopback message traveling across the network is really when there has just been a topology change that has flushed the cam tables. (and BTW, connecting a cable generally leads to a TC, when the port goes forwarding).

Regards,

Francois

  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 4.5 (4 ratings)
Loading.
Konstantin Dunaev Fri, 06/01/2007 - 12:39

let ask me more clear:

MST0 - none VLAN

MST1 - VLAN 1-497

MST3 - VLAN 498

(see att. for better version)

6513 --------------bladeswitch

________--------------________

|________|____________|________|

|1-497(T)|__ ether____|(T)1-497|

|________|____________|________|

|__498(A)|----------->|(A)1____|

\______/______________\_______/

(T) - trunk ports with all VLAN's from MST1

and native VLAN1.

(A)- non-trunk port configured with VLAN on

both side

what will happend with STP and the interfaces if we interconnect now the access ports?

I think will happen following:

1a "distr" will send over etherchannel port the MST BPDU contained MSTI0 and MSTI1.

1b "distr" will send over access port the MST BPDU contained MSTI0 and MSTI3

2a bladeswitch will send over etherchannel port the MST BPDU contained MSTI0 and MSTI1;

2b bladeswitch will send over access port the MST BPDU contained MSTI0 and MSTI1;

2c bladeswitch will forward over access port the MST BPDU that was recieved from distr over etherchannel port.

Will STP discover the loop and block some port?

or will the error "%ETHCNTR-3-LOOP_BACK_DETECTED: Keepalive packet loop-back " on etherchannel and access link on bladeswitch happen?

Konstantin Dunaev Mon, 06/04/2007 - 06:50

HI,

I've reproduced my situation in our test lab and can confirm that MST creates a loop if one interconnects 2 switches with 2 link, one is correctly configured trnk and the second is the access link with wrong configured VLANs on both sides, which belongs to different MST instances.

It seems that in contrast to PVST+ MST doesn't have any tool that can discovery the Port VLAN ID inconsistency.

I think it'S a case for Cisco TAC.

Francois Tallet Mon, 06/04/2007 - 09:06

Hi Konstantin,

I wrote a reply to your initial message last week, but somehow messed up the posting to the forum. Hopefully, I received the message myself and will sent it again right after this one.

I have few questions regarding your test. Can you confirm that you see a loop in your setup? Sure, you get the port errdisable by this loopback message, but is traffic actually looping? If you have a test setup, can you try with the loopback messages disabled (I think it's "no keepalive" in interface config mode)?

I'll try to get more information on how the keepalive is exactly working and will get back to you.

Regards,

Francois

Francois Tallet Mon, 06/04/2007 - 09:07

Hi Konstantin,

About your first question on the pre-STD flag: it is not related to your vlan mapping. Just to the IOS version of the switches. You are running a mix of standard and pre-standard MST bridges in your network, and you have correctly configured "spanning-tree mst pre-standard" on the relevant ports.

There is no problem with moving all the vlans away from MST 0. Sometimes it's convenient to do so because MST 0 is running on every link, including access ports, and its topology can be difficult to understand for engineers who are used to PVST. It seems that you have clearly understood how MST is sending its information so you would not be bothered by that. Anyway, I think the pre-STD flag has nothing to do with the problem here.

About your second question. Connecting the access port in vlan 498 to an access port in vlan 1 in MST is dangerous because MST has no way of detecting this (as you have noticed). PVST is able to detect the issue because it's sending one BPDU per vlan (and the ID of the vlan on which a BPDU is sent is attached in the BPDU for verification purposes). As MST only sends a single BPDU, it cannot detect this problem and you could end up doing a permanent bridging loop this way. Now, in your particular case, I don't think there is a loop because there is only one such "vlan translation" misconfiguration in your network. For example, a frame that would go from the 6513 in vlan 498 would certainly go back to the 6513 in vlan 1, but it would not loop. You've basically merged the two vlans. Of course, as soon as you do a second misconfiguration of this sort (vlan 1 - vlan 498), you're creating a permanent loop. Note that the loop can be more complex, with several vlan translation configuration errors. Something like vlan 1 ---> vlan 498 ---> vlan 5 ----> vlan 1.

Anyway, my problem is that I don't know exactly how this loopback message sent by the blade switch is working. I think the message is sent with the mac address of the port as both a source and destination by the blade switch. That means that the 6513 is going to learn the address of the blade switch and will only forward this frame once (until there is a topology change to clear this entry). The problem I have is that I don't know on which vlan the loopback messages are sent, nor exactly what triggers the inconsistency.

Let me take some guesses here, I'll try to find more information on how this work later.

Suppose that the loopback message is sent on vlan 1 and triggers an inconsistency if it is received on another forwarding port on the same bridge. If you had a loop (2 misconfiguration in your network), it is possible that, when plugging the cable, a topology change is generated in the network and the loopback message is allowed to go through the cat6513 ports and loop back onto the server blades, thus triggering the inconsistency. Another possibility is that you don't have a real loop in your network, but that the loopback message is triggering the inconsistency regardless of the vlan it is received on (I think it would be bad, but I don't know). For example, here, a frame loopback message sent on vlan 1 could be received on vlan 498 on the same bridge and cause the whole trunk to go down.

There is nothing much I can think of so far. HTH.

Francois

Konstantin Dunaev Mon, 06/04/2007 - 11:48

Hi Francois,

thank you very much for your detailed answer!

with my test lab there is one problem - it's a Catalyst 4000 and Catalyst 2950, both with CatOS and I couldn'T find the "keepalive" feature there to get the ports blocked. I'll try tomorrow find a switch with IOS for testing and will try the err-disable scenario.

But anyway the loop is present in my current installation too, I don't know how could I test that this loop is actually loop (I didn't see broadcast storm or MAC address table flapping may be because I don'T have any real hosts in those VLANs), but from output commands I can see it.

I've connected switch "core4" with switch "gt-843-41" with help of 2 links.

2/3(Trunk) <---> 2/49 (Trunk)

2/4 (Access VLAN102) <--> 2/50 (Access VLAN1)

(output from "show trunk", "show spantree mst" is in attachment) here I post just small part from "core4":

swcore4> (enable) sh spantree mst 2/3

Inst State Role Cost Prio VLANs

0 forwarding DESG 1000 32 None

1 forwarding DESG 1000 32 1,100-101,103,105

swcore4> (enable) sh spantree mst 2/4

Inst State Role Cost Prio VLANs

0 forwarding DESG 20000 32 None

2 forwarding DESG 20000 32 102

and from "gt-843-41"

t-843-41> (enable) sh spantree mst 2/49

Inst State Role Cost Prio VLANs

0 forwarding ROOT 20000 32 None

1 forwarding ROOT 20000 32 1,100-101,103,105

gt-843-41> (enable) sh spantree mst 2/50

Inst State Role Cost Prio VLANs

0 blocking ALTR 20000 32 None

1 forwarding DESG 20000 32 1

from here one can mention that all 4 ports, which are somehow has to do with VLAN1 are in "forwarding". Loop is "core4" 2/3 --> "gt-843-41" 2/49 --> "gt-843-41" 2/50 -->"core4" 2/4.

May be for the "real" traffic this kind of loop is not very big issue, it's not good of course, but I don't think it would follow to big connectivity problems, but for "keepalives" I think this loop is enough to put the interface in errdisable mode.

Just as addition: in our case both connection (etherchannel and oracle interconnection) on bladeswitches were put in err-disable status.

From log one could see that the Etherchannel was first and then oracle interconnection.

Attachment: 
Francois Tallet Mon, 06/04/2007 - 13:25

Hi Konstantin,

I'm very clear about what is happening with MST, and this is really expected considering the configuration. Don't bother migrating your switch to IOS, the no keepalive configuration is only relevant to the blade switch in that case. Anyway, my point is that I don't think your traffic is looping (vlan 1 is *not* forwarding in the access port of the core switch). What you basically did is that you merged vlan 1 and vlan 498. Imaging that those two vlans are physical cables: you just connected them to the same hub. The only problem is that, MST is not running through this hub. So everything is fine as long as you only have one connection (the misconfiguration that bridges vlan 1 to vlan 498) between the two vlans. As soon as you introduce a secondary connection between them, then the traffic is going to loop for ever, which will seriously impact your network.

I should be able to find out exactly what the loopback message is doing. I have access to the code, but there are some platform dependent stuff that I cannot interpret easily without help;-) Right now, I suspect that the errdisable is a false positive (not sure though).

Now, whether detecting this is good or bad depend on your point of view. On one hand, it immediately highlighted a configuration error that would have been relatively difficult to troubleshoot. On the other hand, it probably brought down your network:-(

I'll let you know what I find.

Regards,

Francois

Konstantin Dunaev Mon, 06/04/2007 - 14:14

So everything is fine as long as you only have one connection (the misconfiguration that bridges vlan 1 to vlan 498) between the two vlans. As soon as you introduce a secondary connection between them, then the traffic is going to loop for ever, which will seriously impact your network.

hm, I think you're right here.

but as I now understand, it was exactly what you said, we created 2 points of misconfiguration, because we have 2 interconnected distrubution switches and 2 bladeswitches, each connected to one distribution switch, with identical configuration, and actually it was 2 cables that were connected to wrong (not configured) ports on bladeswitches :(.

in this case it seems to be that loopback errdisable status helped us to discovery the problem quick.

but how it looks like? how the switch could see its own loopback keepalive on different interface?

I would be very appreciate if you could find some more information about it.

Francois Tallet Tue, 06/05/2007 - 15:42

So...

Loopback messages are sent periodically on a per physical interface basis. They are sent untagged and even through STP blocked ports, to the destination mac address of the port (and with its source mac address). Of course, because of this interesting particularity, as soon as the neighboring switches have learnt the source mac address of those loopback messages, they are restricted to a single physical link (a switch discards a frame sent to a destination mac address X, if X was learnt on the port on which the frame was received). So the loopback messages are most likely to go through several switches just after a topology change in MST or RSTP (when the cam tables have been flushed).

Now, the errdisable is triggered if a port receives a frame that it has generated. This even if the frame is received through an STP blocked port or on a different vlan (the latter is controversial in my opinion).

To come back to your scenario. As to what you said, you had two ports miswired. The simple way to represent issues related to vlan translation is to split bridges into several virtual bridges, one for each vlan. I've attached a diagram of your network representing the two important vlans: vlan 1 and vlan 498. You'll see: Core-vlan1 and core-vlan498, two bridges representing vlan 1 and vlan 498 on the core switch. Then blade1-vlan1 and blade2-vlan2 etc...

With only one blade switch, you don't create a loop. With two blade switches misconfigured, you can clearly see the loop in the diagram (the virtual diagram is simpler to read than the physical one because the loop is actually spanning across 2 vlans). Note that switches blade 1 and blade 2 won't error disable their trunks, because they don't receive their loopback message back on the port where they sent it. However, all the other blade switches (represented by bladeX on the diagram) will be impacted.

Regards,

Francois

Konstantin Dunaev Tue, 06/05/2007 - 23:35

Note that switches blade 1 and blade 2 won't error disable their trunks, because they don't receive their loopback message back on the port where they sent it. However, all the other blade switches (represented by bladeX on the diagram) will be impacted.

I've checked the logfiles for all blades and you're right! those 2 switches, wherethe wrong port was connected were not in err-disable mode, but all othes were.

But do you mean this can happen only after the MAC table on all switches has been flashed (STP change) and switch core-vlan1 will do flood-unicast for bladeX's loopback massage, because at this time core-vlan1 doesn't know with which port the loopback's source MAC is aasotiated and will nor discard this frame?

And if I've correctly understood you, a switch doesn't update its MAC table (Source MAC <--> physical port) with the information that is in a loopback massage, does it ?

Correct Answer
Francois Tallet Wed, 06/06/2007 - 08:29

Yes.

If a bridge learns a mac address M on a port P (by receiving a frame with M as a source mac address), it means that in the current topology, the device associated with M is off port P. As a result, if the bridge later receives a frame with M as a destination address on port P, it's going to discard it because it knows that the frame will reach its destination without the need to be switched through the bridge.

The loopback message plays a little trick with the learning function of the bridge as its source mac address is the same as its destination mac address. If the cam table does not contain the mac address of the loopback message, it is flooded. After that, it's going to be filtered. So the chance of a loopback message traveling across the network is really when there has just been a topology change that has flushed the cam tables. (and BTW, connecting a cable generally leads to a TC, when the port goes forwarding).

Regards,

Francois

Francois Tallet Wed, 06/06/2007 - 08:54

Yes.

If a bridge learns a mac address M on a port P (by receiving a frame with M as a source mac address), it means that in the current topology, the device associated with M is off port P. As a result, if the bridge later receives a frame with M as a destination address on port P, it's going to discard it because it knows that the frame will reach its destination without the need to be switched through the bridge.

The loopback message plays a little trick with the learning function of the bridge as its source mac address is the same as its destination mac address. If the cam table does not contain the mac address of the loopback message, it is flooded. After that, it's going to be filtered. So the chance of a loopback message traveling across the network is really when there has just been a topology change that has flushed the cam tables. (and BTW, connecting a cable generally leads to a TC, when the port goes forwarding).

Regards,

Francois

Konstantin Dunaev Wed, 06/06/2007 - 11:09

Hi, Francois,

many thank for your time, efforts and brilliant explanation! It was really helpfull.

You saved my reputation of network designer :), because I was near to believe that our network is planned and designed so bad that simply a reconnect of a cable (actually it was done without any notification) lead to the complete black-out.

Francois Tallet Wed, 06/06/2007 - 11:38

Thanks Konstantin,

Well, I learnt about the loopback messages and I had actually fun trying to find out what happened.

The problem you had is a weakness of MST and there is nothing much that can be done to prevent it. Maybe we could create an additional feature that would block a port if the native vlan advertised by a CDP frame received on a port does not match the one locally configured. Because everything comes from this "vlan translation" that was created by the wiring problem. The loopback messages helped finding the issue quickly, but their effect was too dramatic for a safety mechanism in my opinion. On the other hand, it would have been much more difficult to troubleshoot the issue without them.

Regards,

Francois

thorsten.steffen Wed, 06/20/2007 - 03:51

Hello,

my questions are offtopic to the original question of this thread but concern to the cisco keepalives.

So it would be fine if you could me some more information:

- Are keepalives and loopback detection cisco proprietary features or are they common features used also by other switch manufacturers?

- Is the loopback detection the only function of keepalive packets or are they also used for other purposes?

- Is my understanding correct that spanning tree topology changes (with cam table flushing) could cause ports which use keepalives and loopback detection to go in error disabled state?

- If this is the case, is this also the reason that keepalives are disabled by default on fiber and uplink ports since ios versions 12.2SE?

Best regards,

Thorsten

Francois Tallet Wed, 06/20/2007 - 12:30

Afaik, the keepalives are not proprietary. I think they had a use to detect the integrity of some kind of links (old memories, I would have to search google for more on this). However, their use to detect loopback is specific to the 2K/3K ranges of products (it's not even Cisco wide).

The goal of the feature was to detect loopback conditions created by type 1 cabling. For token ring support, the connectors would loop the wires in order to not break the ring when stations were disconnected. When this cabling was used for ethernet, it would send back any frame that it received. That would have the effect of messing up the learning table and potentially loop traffic.

The loopback frames thus detect those ports and shut them down.

I don't know enough on the history of the Cisco products that implement this mechanism. What I know is that on other platforms, we just solved this problem using the spanning tree, which keeps the port in a blocking state if it receives its own BPDUs.

As I mentioned earlier in the thread, because the source and the destination mac of those frames are the same, the learning function of the bridge will prevent them from being switched beyond the first hop.

If you don't have cables looping back frames and if you don't have a briding loop in your network, you should never receive a frame back on the port where you transmitted it.

Even if you have a bridging loop in your network, the transmission of the loopback frames will be most likely prevented by the learning function of the bridges. That means that if you have a bridging loop, you're not likely to detect it with the loopback mechanism before there is an RSTP or MST topology change that clears the cam tables in the network.

I think having this mechanism enabled by default is controversial. Some think that it's an important safety net against network issues. Personnally, I don't like it because:

- it's not standard and it's taking the drastic decision of bringing down ports.

- its original goal is to detect looback cables, not bridging loops. It will actually fail to detect most bridging loops (because of the learning function, and because not all loops sent back a frame to the port which originated it).

Bottom line is that it's been enabled by default for a while. So my guess is that it would require customers repeatedly complaining about negative side effects (not related to a bridging loop or loopback cables) to have this disabled by default.

Regards,

Francois

Actions

This Discussion