SW_MATM-4-MACFLAP_NOTIF

[frede_frede]

Hi

I have a big problem with Vlan flapping, this is my topology

Switch 1 and Switch 2 have the SVI and HSRP configured. Switch 1 is the root bridge for all Vlans. The message vlan X is flapping between port Gi1/0/2 and port Po2 appears on Switch 3. The Po2 is connected to Switch 1.

The port settings in Sw3:

spanning-tree mode pvst

spanning-tree extend system-id

spanning-tree vlan 1-4094 priority 12288

interface Port-channel1

description "Sw2"

switchport trunk encapsulation dot1q

switchport trunk allowed vlan 10,11,12,13

switchport mode trunk

switchport nonegotiate

spanning-tree vlan 1-4094 cost 6

!

interface Port-channel2

description "Sw1"

switchport trunk encapsulation dot1q

switchport trunk allowed vlan 10,11,12,13

switchport mode trunk

switchport nonegotiate

interface GigabitEthernet1/0/21

description "Sw2"

switchport trunk encapsulation dot1q

switchport trunk allowed vlan 10,11,12,13

switchport mode trunk

switchport nonegotiate

channel-group 1 mode passive

!

interface GigabitEthernet1/0/22

description "Sw2"

switchport trunk encapsulation dot1q

switchport trunk allowed vlan 10,11,12,13

switchport mode trunk

switchport nonegotiate

channel-group 1 mode passive

!

interface GigabitEthernet1/0/23

description "Sw1"

switchport trunk encapsulation dot1q

switchport trunk allowed vlan 10,11,12,13

switchport mode trunk

switchport nonegotiate

channel-group 2 mode passive

!

interface GigabitEthernet1/0/24

description "Sw1"

switchport trunk encapsulation dot1q

switchport trunk allowed vlan 10,11,12,13

switchport mode trunk

switchport nonegotiate

channel-group 2 mode passive

In switch 1 and 2 settings are the same except the mode of the channel-group which is active.

I think the problem with "vlan X is flapping" is on the doors of servers that have more than one nic connected to the switch, a misconfiguration can cause the message "Vlan X is flapping Between port Gi1/0/2and port Po2 " and others messages, I say this because this message only appears in configured ports of two VLANs and not in the other two.

Is there any way to prevent such events, even if the problem is the servers?

Thank you for your help

[Peter Paluch]

Hello Frederico,

Your observation is most probably correct. I do not see any obvious problem in your EtherChannel configuration, and I praise very much the fact that you are using LACP to negotiate the creation of your EtherChannels. If the servers are using multiple NICs, it may be necessary to use teaming or other special functionality to make the operating system on these servers to use the NICs properly. I have seen Windows, for example, sending an ARP reply containing MAC address of one NIC through the other NIC, causing some hard-to-debug flooding issues in a switched network. There may be other issues as well.

Sadly, there is nothing to immediately do to prevent these issues from occuring. Where exactly are the servers connected in your topology - which switches and which ports?

Best regards,

Peter

[frede_frede]

HI Peter

The System Administrator has configured teaming. I have 8 servers connected to the switch 3, all with two doors, except one with 4. Only two servers, vlan servers 10 and 11 show problems. The configuration on access doors is the same, except the vlan. The flapping does not appear every day, when there lasts only one minute (minute that it creates instability in the entire network) and then return everything to normal.

It is not strange that the MAC address appears on the access port and in the Po that makes the trunk to the switch1?

Thanks

Fred

[frede_frede]

Another strange thing. In switches 1 and 2, they are two 6500 with FWSM, while I'm receiving the flapping msg  I get the following msg in the 6500 "% SVCLC-5-FWTRUNK: Firewalled VLANs configured on trunks," but the Vlans allowed on the trunks of SW1 SW2 and SW3 are all the same.

Thanks

Fred

[Peter Paluch]

Hello Fred,

Can you update the picture to include the position of one server that exhibits the MAC flapping problem? In the meanwhile, I will try to look up the meaning of the SVCLC message.

Best regards,

Peter

[DuncanM2008]

Hi Fred,

Just to add backing to yours & Peters theories, I've seen the exact message several times on 3750-X series switches and it's normally one of the following:

A) The Servers Networks cards are configured for EtherChannel or LACP (Depending on teaming software) and the switch side of the connection isn't.

B) The Servers aren't fully teamed but are using some form of Transmit Load Balancing (sometimes a default NIC setting with some vendors).

As Peter said though, if you could provide a diagram with the servers in question and how the fit in the topology physically (in terms of connections to which switch)...

HTH

[frede_frede]

First of all thank you for your help.

Here is the update of the topology:

Best regards

Fred

[frede_frede]

Hi Duncan,

when you saw these messages, the flapping was between an access port and a trunk port?

Thanks for the help

Fred

[Peter Paluch]

Hello Fred,

Regarding the SVCLC message, see the following document:

http://www.cisco.com/en/US/docs/ios/12_2sx/system/messages/sm2sx08.html#wp1030512

It states:

Error Message    %SVCLC-5-FWTRUNK : Firewalled VLANs configured on trunks

Explanation    One or more secure VLANs belong to both secured and ordinary trunks. This  configuration may compromise the security of the secure VLANs.

Recommended Action    Change the trunk or the secure VLAN configuration so that they do not overlap.  Enter either the switchport trunk allowed vlan command to change trunk configuration or the  firewall vlan group command to change the secure VLAN configuration.

Most other documents I have been able to find on Google state that this is a cosmetic message, simply informing you that the VLANs that have been assigned to the FWSM are also allowed on some trunks, and may therefore bypass the FWSM. If that is true that it seems that this message is not related to our issue although I am not willing to say that with a definitive certainty.

Fred, regarding the MAC address flapping message... If it is indeed found flapping between Gi1/0/2 and Po2 then it means that frames sourced from the same MAC address are entering both Gi1/0/2 and Po2 ports. How could that happen? Either there is a loop somewhere in the network, or there are two devices in the network that somehow use the same MAC address. It would be interesting to see if the STP still keeps redundant ports as blocking during periods of flapping. Is there any other device in the network, including servers, that could theoretically loop frames received from the server and put them back into network? Also, what is the exact MAC address that is reported flapping?

Sorry for not moving this issue any further... I am still trying to get a firm grasp of what is happening.

Best regards,

Peter

[frede_frede]

Peter very grateful for your help

I'm also very surprised. I also thought that this behavior was derived from the STP, but if the source of the problem is STP,  this behavior should happen in all Vlans and not just in two?

These are the messages, but the time window of this behavior is short,

2012-06-13 09:59:44   SW_MATM-4-MACFLAP_NOTIF   Host 0015.1728.420a in vlan 11 is flapping between port Po2 and port Gi1/0/13

2012-06-13 09:59:43   SW_MATM-4-MACFLAP_NOTIF   Host 0030.13e5.34f2 in vlan 10 is flapping between port Po2 and port Gi1/0/17

2012-06-13 09:59:43   SW_MATM-4-MACFLAP_NOTIF   Host 0030.13e5.34f3 in vlan 10 is flapping between port Po2 and port Gi1/0/19

2012-06-13 09:59:43   SW_MATM-4-MACFLAP_NOTIF   Host 0030.13e5.34f1 in vlan 10 is flapping between port Po2 and port Gi1/0/20

2012-06-13 09:59:43   SW_MATM-4-MACFLAP_NOTIF   Host 0030.13e5.34f0 in vlan 10 is flapping between port Po2 and port Gi1/0/18

2012-06-13 09:59:43   SW_MATM-4-MACFLAP_NOTIF   Host 0015.1728.420c in vlan 11 is flapping between port Po2 and port Gi1/0/13

I put one port of the server in vlan 11 in shutdown mode, currently it has only one active. The server in vlan 10 continues with 4 doors.

Best regards,

Fred

[Vivek Ganapathi · Accepted solution]

Hi Fred,

I would suggest you to check for the TCNs for those VLANs (command : show spanning-tree vlan 10 detail). To may want to see the incrementing numbers of TCNs. It even mentions the last topology change notification from which interface.

You may need to monitor the above for a day or so to figure out if it's related to STP.

This issue can be due to multiple factors like UDLD, Server-side LB mechanism etc. Hit each of the points.

Regards

Vivek

[frede_frede]

Hi Vivek,

I'm going to check TCN to see if the value of TCN is modified.

If so, what can I do to protect the STP, I already modified the cost.

Regards

Fred

I mistakenly put the question as answered, but this is not, how can I pull this off?

[Vivek Ganapathi]

Hi Fred,

Unfortunately, you cannot pull it off. We will continue discussion on this.

Ok, when you are seeing the MAC flap error, you can just check the TCN value. If it's getting modified trace down to the source of the problem. You may not be able to do much with the STP, but it would help you reach to the source of the problem (May be a Unidirectional issue).

But do re-verify the LB algo on the server side as well. If the box is solaris & the configuration is left to defaults, its known to cause problems. There are some parameters on it which needs to be modified.

Lets work on this. Keep us on posted on whats happening.

Regards

Vivek

[frede_frede]

Hi Vivek,

Yesterday there was no problem, but I'm vigilant.

The servers are not Solaris, but next week the server guy is returning from vacations and I can have his help too

Regards

Fred

[Vivek Ganapathi]

Great fred. Will wait for your next update.