Cisco Support Community
cancel
Showing results for 
Search instead for 
Did you mean: 
Announcements

Welcome to Cisco Support Community. We would love to have your feedback.

For an introduction to the new site, click here. If you'd prefer to explore, try our test area to get started. And see here for current known issues.

New Member

Invalid MAC address endlessly flapping between two ports

Hi guys,

I've hit an issue that I can't get to the end of, and thought that you might be able to help...

I have an invalid MAC address that is flapping and continously looping between in aVLAN of one of our remote sites LAN. The mac address is

6000.86dd.6000 and I can't find it on any access port of any switch in my network, with no idea of what to generate. I mention that I have rapid-pvst enabled on all switches and that the STP topology is stable, with the proper switches blocking the proper ports.

The core stack CPU is at 90% CPU with the below process being the main resource drain:

69    290298741597995435         18 13.25% 10.89% 10.59%   0 HLFM address lea

The log buffer is full of these messages, but only on the core switch, as the other access switches see that mac on the uplink ports only, and no flapping is detected.

Dec 20 19:38:39 PHT: %SW_MATM-4-MACFLAP_NOTIF: Host 6000.86dd.6000 in vlan 75 is flapping between port Gi1/0/25 and port Gi2/0/26

Dec 20 19:38:54 PHT: %SW_MATM-4-MACFLAP_NOTIF: Host 6000.86dd.6000 in vlan 75 is flapping between port Gi1/0/25 and port Gi2/0/26

Dec 20 19:39:09 PHT: %SW_MATM-4-MACFLAP_NOTIF: Host 6000.86dd.6000 in vlan 75 is flapping between port Gi1/0/25 and port Gi2/0/26

Dec 20 19:39:24 PHT: %SW_MATM-4-MACFLAP_NOTIF: Host 6000.86dd.6000 in vlan 75 is flapping between port Gi2/0/26 and port Gi1/0/25

The core switch is a 2 3750 stack and all the access switches are 2960S stacks (3-4 switches per stack).

For a better understanding of this issue, I have attached a network diagram and some command outputs.

I already tried clearing the cam tables simlutaneously, but with no effect (the siwtches forward frames a lot faster than me sending the clear commands from the ssh sessions).

I would appreciate any idea for solving this issue.

Everyone's tags (3)
17 REPLIES
New Member

Invalid MAC address endlessly flapping between two ports

Hi Razvan,

I seen this often on misconfigured ports. The trunk forward the packets and on the other side they are received as "not trunk packets", and thus the same mac seem to enter the Vlan at 2 places.

It can also be produced by creating a monitor session on one switch and send the packet to an edge port on another switch, that will cause massive duplicate packets. But I don't expect you to have that setup in a production network.

Gert

New Member

Re: Invalid MAC address endlessly flapping between two ports

Hi Gert,

You would be right on that one, I don't have this scenario on this network. Would you guys know where could this MAC originate from? It doesn't seem to be a legitimate MAC address to me...

New Member

I know this is old, but I'd

I know this is old, but I'd say it's a 'bond' which has either gone wrong or one of the NICs is faulty.

Re:Invalid MAC address endlessly flapping between two ports

Hello

This output suggests a stp.loop


Start by checking.your topology diagram then begining from the stp root switch progess to each switch which has a trunk interconnect

Evenually you should see a port that should be in a blocking state causing the loop

Sh interface.trunk
Sh spanning-tree vlan 75

Res
Paul

Sent from Cisco Technical Support Android App

Please don't forget to rate any posts that have been helpful. Thanks.
New Member

Hi, we are currently

Hi, we are currently experiencing this exact same problem too. Anyone knows the solution to this?

New Member

Answer: There is a bug on the

Answer: There is a bug on the 3750X IOS. The IOS used was 15.0(2)SE4. We upgraded it to SE7.

The bug states that these malformed IPv6 packets are being forwarded by blocking ports. That's why the switches suddenly has a high CPU.

New Member

Hello,It happened again, but

Hello,

It happened again, but this time with a different Invalid MAC address 48:22:86:dd:60:00. We are engaged with Cisco TAC but they are currently unable to find this MAC yet.

This just appears on the logs of the core switch with it appearing on the uplinks. From the access switches, it doesn't point to an access port.

The odd thing about this is the MAC address is an invalid one.

 

New Member

Do you happen to have a

Do you happen to have a wireshark capture of these packets?
 

New Member

Hi,

Hi,

Have you solved the issue? I need your help.

Re:Invalid MAC address endlessly flapping between two ports

Hello

This output suggests a stp.loop


Start by checking.your topology diagram then begining from the stp root switch progess to each switch which has a trunk interconnect

Evenually you should see a port that should be in a blocking state causing the loop

Sh interface.trunk
Sh spanning-tree vlan 75

Res
Paul

Sent from Cisco Technical Support Android App

Please don't forget to rate any posts that have been helpful. Thanks.
New Member

Re: Invalid MAC address endlessly flapping between two ports

Hi pdriver,

I do not understand how a port in a blocking state can cause the loop. The Blocking state is supposed to break the loop, right? I have initially attached the output of the "Sh spanning-tree vlan 75" command. STP topology seems to have converged well and no loops seem to exist on the VLAN 75 STP topology.

The blocked ports for this vlan are on 29stk2 (Gi2/0/49) and 30stk2(Gi2/0/49):

---------------------------------------------------------------------------------------------------------------------------------------------------

mnla_29stk2#sh spanning-tree vlan 75

VLAN0075

  Spanning tree enabled protocol rstp

  Root ID    Priority    24651

             Address     8cb6.4f76.5780

             Cost        4

             Port        49 (GigabitEthernet1/0/49)

             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    32843  (priority 32768 sys-id-ext 75)

             Address     b862.1fed.7d80

             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

             Aging Time  300 sec

Interface           Role Sts Cost      Prio.Nbr Type

------------------- ---- --- --------- -------- --------------------------------

Gi3/0/2             Desg FWD 19        128.110  P2p Edge

Gi3/0/47            Desg FWD 4         128.155  P2p

Gi3/0/48            Desg FWD 4         128.156  P2p

Gi1/0/17            Desg FWD 19        128.17   P2p Edge

Gi1/0/49            Root FWD 4         128.49   P2p

Gi2/0/37            Desg FWD 19        128.91   P2p Edge

Gi2/0/49            Altn BLK 4         128.103  P2p

---------------------------------------------------------------------------------------------------------------------------------------------------

mnla_30stk2#sh spanning-tree vlan 75

VLAN0075

  Spanning tree enabled protocol rstp

  Root ID    Priority    24651

             Address     8cb6.4f76.5780

             Cost        4

             Port        49 (GigabitEthernet1/0/49)

             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    32843  (priority 32768 sys-id-ext 75)

             Address     b862.1fe3.9a00

             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

             Aging Time  300 sec

Interface           Role Sts Cost      Prio.Nbr Type

------------------- ---- --- --------- -------- --------------------------------

Gi3/0/31            Desg FWD 4         128.139  P2p

Gi3/0/48            Desg FWD 4         128.156  P2p

Gi1/0/49            Root FWD 4         128.49   P2p

Gi2/0/49            Altn BLK 4         128.103  P2p

---------------------------------------------------------------------------------------------------------------------------------------------------

Re: Invalid MAC address endlessly flapping between two ports

Hello

I am sorry you misunderstood my last post -when I stated " Evenually you should see a port that should be in a blocking state causing the loop "


I was saying a port that should be in a blocking state is currently forwarding causing the loop.

Most of the times this is caused by a misconfiguration of a access port and attaching a unwarranted switch/hub to the network introducing a loop

Have you also checked for any span sessions or the not so nice stp bpdufilter command applied to any access ports that now have a switch/or hub attached?

Res
Paul

Sent from Cisco Technical Support iPad App

Please don't forget to rate any posts that have been helpful. Thanks.
New Member

Re: Invalid MAC address endlessly flapping between two ports

Yes Paul, soory for the missunderstanding. The loop is only contained in VLAN 75.

There is only one span session that is involving other VLANs, not vlan 75 (and it is configured on the core switch only). It is not configured for any of the trunk ports.

mnla_core_1#sh run | inc monitor session

monitor session 1 source vlan 50 - 65

monitor session 1 destination interface Gi1/0/24

monitor session 1 destination interface Gi2/0/14

monitor session 2 source vlan 50 - 65

monitor session 2 destination remote vlan 505

mnla_core_1#sh cdp ne Gi1/0/24

Capability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge

                  S - Switch, H - Host, I - IGMP, r - Repeater, P - Phone

Device ID        Local Intrfce     Holdtme    Capability  Platform  Port ID

mnla_core_1#sh cdp ne Gi2/0/14

Capability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge

                  S - Switch, H - Host, I - IGMP, r - Repeater, P - Phone

Device ID        Local Intrfce     Holdtme    Capability  Platform  Port ID

mnla_core_1#sh run int Gi1/0/24

Building configuration...

Current configuration : 122 bytes

!

interface GigabitEthernet1/0/24

description orecx-monitor-new

switchport access vlan 10

speed 1000

duplex full

end

mnla_core_1#sh run int Gi2/0/14

Building configuration...

Current configuration : 118 bytes

!

interface GigabitEthernet2/0/14

description orexc-monitor

switchport access vlan 10

speed 1000

duplex full

end

Bronze

Re: Invalid MAC address endlessly flapping between two ports

Hello Razvan,

Please attach a show logg from 29stk2  and 30stk2

Haihua

New Member

Re: Invalid MAC address endlessly flapping between two ports

I'm a coworker of Razvan's.

the log entries for those two swtiches are pretty clean..  For example, today, there are only entries regarding power to IP phones and logins. 

I'd post them but i've got cut and paste issues at the moment.

I did find that one of the trunk/uplink ports was configured with portfast which I removed.  Not that it mattered because it was only portfast, not trunk portfast.

I also removed vlan 75 from the trunk on both sides of one of the connections but the errors continue.

I think the key is if we can identify the where/why on the weird mac address it is seeing coming through the two links.

Another thought I had was that  somehow our wireless access points were involved.  They are connected to these two switches.  but I removed vlan 75 from their uplink config, also with no impact.

HelloThe mac flap is between

Hello

The mac flap is between switches 29stk1 and 30stk2 so looking at you topology they are not connected and shouldnt be either, but obviously the core is reporting seeing it from either switch,

As I stated in a previous post look for stp bpdu-filter applied to a port or a monitor port that is not being used and now is an access port
also sh int trunk check the inferaces that are forwarding on this vlan 75 - between your cloest switches

Lastly if this issue isnt occurring all the time, beginning at the core you can trace the existing port this mac is located on by
sh ip arp | in 6000.86dd.6000 
sh mac-address address 6000.86dd.6000 
-

res
Paul

 

Please don't forget to rate any posts that have been helpful. Thanks.
New Member

Hi,we have had the same issue

Hi,

we have had the same issue on 3 separate occasions on our network.

CPU of the router (in our case a 4900M) would go to 100%

debugging on the switch would show packet like fe.

Index 15:

194 days 1:16:48:536508 - RxVlan: 7, RxPort: Te1/8

Priority: Normal, Tag: Dot1Q Tag, Event: Input Acl Fwd, Flags: 0x40, Size: 516

Eth: Src 60:36:86:DD:60:36 Dst 86:DD:60:00:86:DD Type/Len 0x86DD

Remaining data:

 0: 0x60 0x36 0x86 0x0  0x0  0x36 0x0  0x0  0x0  0x36

10: 0x0  0x1  0xFE 0x80 0x0  0x1  0xFE 0x80 0x0  0x1

20: 0xFE 0x80 0x0  0x1  0xFE 0x80 0x11 0x1  0xFE 0x0

30: 0x11 0x0  0x0  0x0  0x0  0x0  0x0  0x0  0x0  0x0

40: 0x0  0x0  0x0  0x0  0x0  0x0  0x0  0x0  0x2  0x60

 

 

As you can see, both source and desination are somewhat peculiar and something seems to point at ipv6 due to the 86dd  (even though there is no ipv6 config on that routers L3 interface)

The destination addresses seem to be 86:DD:60:00:86:DD or 86:DD:60:84:86:DD

We place a "mac address-table static 86dd.6000.86dd vlan x drop" on the router and then the packets went away

not a problem of loops, but something specifically tied to these strange mac-adresses.

We suspect either a funny application that messes things up, or something happening on the router.

It would bounce even on a single link (no loop environment), looking at the mac-tables you would conclude they send out the packet back on the interface on which they received it, which normally can't be the case.

 

In our last case, this caused a problem with hardphones (internal switch) and authentication.

As somehow it would trigger packets responses or something with mac-adres sources like 6084.86dd.xxxx

 

 

11696
Views
0
Helpful
17
Replies