Missing MAC Address

Answered Question
Aug 17th, 2010
User Badges:

Hi,


I've got four switches in a line. On the fourth, an AP is connected. A PC, associated with this AP, sends ethernet frames through the switches, but its MAC address doesn't show up in the MAC tables like this:


--- snip ---

Switch1#ping 10.149.251.33


Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.149.251.33, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/3/4 ms
Switch1#
Switch1#sh ip arp | in 10.149.251.33
Internet  10.149.251.33          93   0016.6f75.861e  ARPA   Vlan251
Switch1#sh mac add add 0016.6f75.861e
No entries present.

--- snap ---


Only the switch which the access point is connected to knows the MAC address.


The port in the first switch leading to the host in question is configured like this:


--- begin config ---

interface GigabitEthernet1/45
switchport access vlan 7
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 7,251
switchport mode trunk
no snmp trap link-status
spanning-tree portfast
end

--- end config ---


Other MAC addresses ARE present on the port in VLAN 251


Any suggestions?


Thanks

Correct Answer by Peter Paluch about 6 years 9 months ago

Peter, Milan,


Indeed, we have found the source of Peter's problems: a workstation connected into the same VLAN by two distinct interfaces.


Peter, connecting a station with two NICs into the same VLAN (and thus in the same IP subnet) is usually calling for trouble like this one. Common operating systems like Windows or Linux usually do not have any intelligent mechanisms to decide which interface they should use when communicating, and it can be almost a random play to find out which interface is actually answering to the ARP requests and which one is used to send packets.


What I believe happened is that when the laptop sent and ARP message to the network (either its own ARP Request or an ARP Reply), it inserted its IP address and the MAC address of the wireless NIC into the ARP message body. However, that entire ARP message was then sent out the wired NIC and that is the source of confusion - an ARP message containing the MAC address of a particular interface, encapsulated in a frame that went out through a different interface! That is why the ARP cache and the MAC address tables on all switches got "desynced". Blame the operating system on the laptop!


Actually, this could be very interesting to watch in Wireshark and confirm whether this is really the case. Wanna do some tests on the laptop?


In any case, as Milan has pointed out, it would be the best if the VLAN for the wireless clients was separated from the wired VLAN. At least, do not connect a PC with two interfaces into the same VLAN.


One comment to your configuration: on ports configured as static trunks using the switchport mode trunk, the command switchport access vlan X is unused and ignored. As it can be confusing to see that command on a trunk, I suggest removing it from all ports that are configured similarily, to reduce the ambiguosity of the configuration.


Best regards,

Peter

Correct Answer by milan.kulik about 6 years 9 months ago

Hi,


it's clearer now.


IMHO, it's a bad practice to use the same VLAN and subnet for wireless and cable connection :-(

You might get confused then easily as happened in your case.


Connecting both cable and wireless on your PC creates a loop in VLAN251 in fact.


It's possible you've got bridging enabled on your PC.

And STP running on that might shutdown the wireless port.

But it might be considered Up by the IP stack still, i.e., replying to Pings received by the Ethernet interface.

My understanding what could happen is:

When you try to Ping your PC from Switch1, the switch sends and ARP Request broadcast to VLAN251.

The PC receives it via the cable and as it knows there's 10.149.251.33 configured on the wireless interface, it replies.

But the ARP Reply is sent from the PC's Ethernet interface.

So the Switch1 knows the MAC address of the PC's wireless interface (incuded inside the ARP Reply packet).

But as the ARP Reply was sent out from the Ethernet interface, the Switch1 has never received any frame sent out from the PC's wireless interface and does not have the wireless MAC address in the MAC address table.

And replies to Pings are received through the cable part of your LAN, again.


I know this theory does not explain why Switch3 knows the MAC address of the wireless interface.

It's possible the Switch3 is breaking the STP loop and the PC is replying to Pings from the Ethernet interface due to a default route.

Or something even more complicated.


But definitely, it's not good to connect a PC to a VLAN by two interfaces configured for the same subnet.


IMHO, if you disconnect your PC Ethernet port from the LAN and connect it via wireless only, everything should work a standard way.


HTH,

Milan

  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 5 (2 ratings)
Loading.
Peter Paluch Tue, 08/17/2010 - 05:04
User Badges:
  • Cisco Employee,

Hi,


Interesting. What exact type of switches and the IOS version are you using? Also, if I understand your description correctly, the AP is connected to a trunking port, right? Are you using multiple SSIDs/VLANs on your AP? In which VLAN is the PC's MAC address learned on the first switch?


Best regards,

Peter

Nagaraja Thanthry Tue, 08/17/2010 - 05:51
User Badges:
  • Cisco Employee,

Hello,


Are you seeing packets destined to that PC on other ports as well? Can you

place a sniffer on VLAN 251 and see if the switch is flooding the traffic?

If it is, then it could be either that the switch is out of resources or the

MAC address table is corrupt. Can you post the output of "show platform tcam

utilization" here?


Regards,


NT

Peter Fiers Tue, 08/17/2010 - 21:45
User Badges:

Hello gentlemen,


sorry for the confusion. The exact picture looks like this:


Switch1 WS-C4948 (cat4500-ENTSERVICESK9-M), Version 12.2(54)SG

      |

      |

Switch2 WS-C2950G-24-EI (C2950-I6Q4L2-M), Version 12.1(22)EA13

      |

      |

Switch3 WS-C2950G-24-EI (C2950-I6Q4L2-M), Version 12.1(22)EA13

      |

      |

     AP AIR-AP1121G-E-K9 (C1100-K9W7-M), Version 12.3(8)JA2

      |

      |

     PC 0016.6f75.861e 10.149.251.33


All uplinks are statically configured as trunks and have the VLANs 7 (switch mgmt) and 251 (clients) allowed.

The AP has dot1q subinterfaces f0.7 and f0.251. It's managed through the BVI1 which is associated with f0.7. In addition, there is one SSID in the AP which is assigned to the bridge-group 251, just like f0.251. The PC is associated with this SSID. Switch3 learns the MAC address on its port leading to the AP in VLAN 251. By configuration, the frames from the PC have to be tagged with dot1q 251.



None of the first two switches knows the MAC address of the PC. Switch1 in fact seems to flood the frames destined to the MAC address 0016.6f75.861e. We registered DHCP Acks for this host on another port. The MAC address table of the switch isn't saturated, however:


--- begin command output ---

Switch1#sh mac address-table count

MAC Entries for all vlans:

Dynamic Unicast Address Count:                  123

Static Unicast Address (User-defined) Count:    0

Static Unicast Address (System-defined) Count:  9

Total Unicast MAC Addresses In Use:             132

Total Unicast MAC Addresses Available:          32768

Multicast MAC Address Count:                    14

Total Multicast MAC Addresses Available:        32768

--- end output ---


Switch2 doesn't seem to have problems with space for MAC addresses, either:


--- begin command output ---

Switch2#sh mac-add count


Mac Entries for Vlan 251:

---------------------------

Dynamic Address Count  : 9

Static  Address Count  : 0

Total Mac Addresses    : 9


Mac Entries for Vlan 7:

---------------------------

Dynamic Address Count  : 7

Static  Address Count  : 0

Total Mac Addresses    : 7


Total Mac Address Space Available: 8174

--- end output ---



The command "show platform tcam utilization" isn't available in the switches.


Btw, there is no possibility for the PC for roaming between two access points as there is no other AP there. Neither is link redundancy, the frames from the PC can take only one path.


Thanks,

milan.kulik Tue, 08/17/2010 - 23:54
User Badges:
  • Red, 2250 points or more

Hi,


hasn't anybody changed anything in your LAN recently?

(Like removing  an L3 IP interface configured  in VLAN 251 from your switch1?)

Can you try traceroute from your switch1 to the PC?

I can imagine the packet is routed somewhere between VLAN 7and 251.

But can't explain how the ARP entry would appear in your switch1.


BR,

Milan

Peter Fiers Wed, 08/18/2010 - 01:09
User Badges:

Hello Milan,


traceroute shows one hop as expected. No changes in the VLAN configuration were made since the switches are up and running (after the last maintenance downtime).


Do you have any idea how to debug / troubleshoot MAC address table changes / inconsistencies? Or if I can capture / debug ethernet frames as I do with IP packets?


Thanks,

Peter Paluch Wed, 08/18/2010 - 07:08
User Badges:
  • Cisco Employee,

Hello,


A couple of questions:


  1. Are you perhaps using the VTP Pruning in your network? Verify it using the show vtp status command on each switch.
  2. Are you running any STP on your network? Can you verify its correct behavior? Are all trunks Forwarding on both ends? Use the show spanning-tree commands to verify that.
  3. Can you verify using the show int trunk that the VLANs 1, 7 and 251 are properly enabled and not pruned from the trunks? No native VLAN mismatches?
  4. Is the topology you have indicated earlier the complete topology? Just three switches in a row - no possible other ways, no other loops, no detours, perhaps even through wireless?
  5. Is it possible that the MAC address table aging time is reduced? Try looking on the show mac address-table aging-time
  6. Is it possible that there is a steady topology change detected under your STP? Try looking closely at the show spanning-tree detail command. In particular, look for the fields  "Number of topology changes" and "Number of transitions to forwarding state". In a stable network, the numbers should be low and should not increase over time.
  7. Are you using RSPAN in your network? Is it possible that the VLAN 251 has been inadvertently declared as RSPAN VLAN?


Seeing an DHCP ACK message on a different port is normal under circumstances - this message may be broadcasted if the client requests the server to do so.


Best regards,

Peter

Peter Fiers Wed, 08/18/2010 - 23:13
User Badges:

Hi Peter,


VTP isn't in use (operating mode: transparent) on the switches as is in the whole network.
Switch1 is running RSTP, the other ones behind it PVST. The topology is stable. There were no topology changes since starting the switches. All ports show one or two transitions to forwarding state. Switch1 is not the root, but the root ports on the other switches point to it. All trunks are in the forwarding state.
The trunks are identically configured:
Mode: on
Encap.: dot1q
Native VLAN: 1
Allowed VLANs: 7,251
The only difference in the port configurations is the access VLAN. Some have 7 some 1 and one 251, but it shouldn't be relevant as the ports do operate in trunk mode.
MAC address aging time is overall 300 s.
No RSPAN in use.


I think Switch1 is flooding the frames for the MAC address in question because an other switch connected to it (and probably not mentioned so far) gets them as following log of DHCP Snooping on that switch shows. It is the unicast address that appears in 'MAC da':
--- begin log ---
Aug 16 16:50:26.407 cet: DHCP_SNOOPING: process new DHCP packet, message type: DHCPACK, input interface: Gi0/1, MAC da: 0016.6f75.861e, MAC sa: 001b.5398.1b7f, IP da: 10.149.251.33, IP sa: 10.102.32.181, DHCP ciaddr: 10.149.251.33, DHCP yiaddr: 10.149.251.33, DHCP siaddr: 10.102.32.181, DHCP giaddr: 0.0.0.0, DHCP chaddr: 0016.6f75.861e
--- end log ---


The physical topology is a bit more complex, but VLAN 251 is very simple. It shows a star topology with Switch1 in the middle. As far as I know, there are no other APs there, either.


What about debugging the ethernet frame traffic and/or MAC address learning on the switch? I only don't know how...


Thanks,

Peter Paluch Thu, 08/19/2010 - 00:34
User Badges:
  • Cisco Employee,

Hi Peter,


I think it would be very helpful if you could provide us with a diagram of your physical topology. The issue is getting convoluted and the information we've exchanged so far does not bring any light in it.


Also, do you believe you could post the configuration of your devices (without sensitive info, of course)?


To address some of your answers:


  1. You wrote that "Switch1 is running RSTP, the other ones behind it PVST." Why is that? If all switches support RSTP then it is a very bad habit to combine different versions of STP in your network. According to the info about your switches and IOSes, all those switches should support RSTP.
  2. You stated: "The only difference in the port configurations is the access VLAN. Some have 7 some 1 and one 251, but it shouldn't be relevant as the ports do operate in trunk mode." If I understand you correctly here then you are suggesting that ports operating in trunking mode differ in their access VLAN - is that so? Are we talking about ports between the switches or the ports connected to end stations? Having an opportunity to look into your configurations would help very much.
  3. Regarding the RSPAN VLAN, please double check its (non)existence with the command show vlan remote-span
  4. Regarding debugging the MAC address table management, try using the commands debug matm and debug platform matm all (the latter may not be supported on all platforms). Also you could try using the MAC notification feature described in the following URL.


http://www.cisco.com/en/US/docs/switches/lan/catalyst3560/software/release/12.2_55_se/configuration/guide/swadmin.html#wp1160164


Best regards,

Peter

Peter Fiers Thu, 08/19/2010 - 22:34
User Badges:

Hi Peter,


sorry for that, but it took some time to check out the situation on-site.


What I've found out is that, while there are no switching loops and link redundancies (not even via wireless), the PC is connected to the network twice: via wireless LAN and through the cable as depicted in the diagram attached. (The network is actually much more complex than that, but the diagram shows the relevant part.) The cable connection works as expected, the MAC address of the ethernet interface is known by all switches in VLAN 251. Probably we should focus on this fact because it's an anomaly per se and the configs of the switches may not show anything unusual. I'll try to make some tests like pulling the cable or sniffing on the interfaces. What I think is that the PC advertises its MAC address of the wireless interface, but these advertisements never go out of the wireless interface, instead they are encapsulated in packets leaving the PC through the cable interface. There are some issues with this theory, however: Why does Switch3 know the MAC address of the PC and how is it possible to ping the IP of the wireless interface (10.149.251.33)?


Unfortunately, it's not trivial for me to post the configs of the switches as I would face authorization issues and currently I'm not able to go them through looking for private items.


As to the STP issue, the reason might be ignorance, indeed. I'll propose a correction of that.


It may help if I post at least the trunk configurations what I can do without much effort:


Switch1

======

interface GigabitEthernet1/45
description #2404290
switchport access vlan 7
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 7,251
switchport mode trunk
no snmp trap link-status
spanning-tree portfast
end


interface GigabitEthernet1/41
description #2029263
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 7,251,432,632,633,832,833
switchport mode trunk
logging event link-status
logging event trunk-status
end


interface GigabitEthernet1/39
description xxx
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 7,251,432,832,833
switchport mode trunk
end


Switch2

======

interface GigabitEthernet0/1
switchport access vlan 7
switchport trunk allowed vlan 7,251
switchport mode trunk
rmon collection stats 26 owner root
end


interface GigabitEthernet0/2
switchport trunk allowed vlan 7,251
switchport mode trunk
rmon collection stats 27 owner root
end


Switch3

======

interface GigabitEthernet0/1
switchport trunk allowed vlan 7,251
switchport mode trunk
rmon collection stats 26 owner root
end


interface FastEthernet0/24
description AP01
switchport access vlan 251
switchport trunk allowed vlan 7,251
switchport mode trunk
no logging event link-status
no snmp trap link-status
rmon collection stats 25 owner root
spanning-tree portfast
end


interface FastEthernet0/3 --> It's an access port.
switchport access vlan 251
switchport mode access
no logging event link-status
no snmp trap link-status
rmon collection stats 4 owner root
spanning-tree portfast
end


Thanks,

Correct Answer
milan.kulik Thu, 08/19/2010 - 23:39
User Badges:
  • Red, 2250 points or more

Hi,


it's clearer now.


IMHO, it's a bad practice to use the same VLAN and subnet for wireless and cable connection :-(

You might get confused then easily as happened in your case.


Connecting both cable and wireless on your PC creates a loop in VLAN251 in fact.


It's possible you've got bridging enabled on your PC.

And STP running on that might shutdown the wireless port.

But it might be considered Up by the IP stack still, i.e., replying to Pings received by the Ethernet interface.

My understanding what could happen is:

When you try to Ping your PC from Switch1, the switch sends and ARP Request broadcast to VLAN251.

The PC receives it via the cable and as it knows there's 10.149.251.33 configured on the wireless interface, it replies.

But the ARP Reply is sent from the PC's Ethernet interface.

So the Switch1 knows the MAC address of the PC's wireless interface (incuded inside the ARP Reply packet).

But as the ARP Reply was sent out from the Ethernet interface, the Switch1 has never received any frame sent out from the PC's wireless interface and does not have the wireless MAC address in the MAC address table.

And replies to Pings are received through the cable part of your LAN, again.


I know this theory does not explain why Switch3 knows the MAC address of the wireless interface.

It's possible the Switch3 is breaking the STP loop and the PC is replying to Pings from the Ethernet interface due to a default route.

Or something even more complicated.


But definitely, it's not good to connect a PC to a VLAN by two interfaces configured for the same subnet.


IMHO, if you disconnect your PC Ethernet port from the LAN and connect it via wireless only, everything should work a standard way.


HTH,

Milan

Peter Fiers Thu, 08/19/2010 - 22:57
User Badges:

Hi Peter,


it's an update to my last post.


Neither of the debug commands recommended exist on the 4948. Instead, I've found the command 'debug platform packet'. The attached output seems to approve my theory as you can see only packets coming from the cable interface (or at least with the MAC address of it) whether you ping the IP of the cable interface or that of the wireless interface. I think we've got the origin of the problem, only the explanation is missing...


The debug commands I used:

deb plat pack 000b.97a2.3db1 rec
deb plat pack 0016.6f75.861e rec


Thanks,

Correct Answer
Peter Paluch Fri, 08/20/2010 - 01:23
User Badges:
  • Cisco Employee,

Peter, Milan,


Indeed, we have found the source of Peter's problems: a workstation connected into the same VLAN by two distinct interfaces.


Peter, connecting a station with two NICs into the same VLAN (and thus in the same IP subnet) is usually calling for trouble like this one. Common operating systems like Windows or Linux usually do not have any intelligent mechanisms to decide which interface they should use when communicating, and it can be almost a random play to find out which interface is actually answering to the ARP requests and which one is used to send packets.


What I believe happened is that when the laptop sent and ARP message to the network (either its own ARP Request or an ARP Reply), it inserted its IP address and the MAC address of the wireless NIC into the ARP message body. However, that entire ARP message was then sent out the wired NIC and that is the source of confusion - an ARP message containing the MAC address of a particular interface, encapsulated in a frame that went out through a different interface! That is why the ARP cache and the MAC address tables on all switches got "desynced". Blame the operating system on the laptop!


Actually, this could be very interesting to watch in Wireshark and confirm whether this is really the case. Wanna do some tests on the laptop?


In any case, as Milan has pointed out, it would be the best if the VLAN for the wireless clients was separated from the wired VLAN. At least, do not connect a PC with two interfaces into the same VLAN.


One comment to your configuration: on ports configured as static trunks using the switchport mode trunk, the command switchport access vlan X is unused and ignored. As it can be confusing to see that command on a trunk, I suggest removing it from all ports that are configured similarily, to reduce the ambiguosity of the configuration.


Best regards,

Peter

Peter Fiers Fri, 08/20/2010 - 02:21
User Badges:

Thank you, gentlemen, I think, we've got the solution. I'll try to get the owner of the PC to make traces of network traffic on both of the interfaces and will then update this discussion. Btw, double-connecting the PC was obviously not his intention. Actually, there is a policy in place telling only one interface should be connected at any time on a PC, but hmm...


Best regards,


Peter

Peter Fiers Mon, 09/13/2010 - 04:12
User Badges:

Hi Peter,


I've got the trace now. In the attachment, you can see two packets: DHCP Req and DHCP Ack, captured on the ethernet interface of the notebook. You can learn the MAC address of this interface from the first packet: 00:0b:97:a2:3d:b1. Note that the interface uses the IP address of the wireless interface (10.149.251.33). Its own IP is 10.149.251.35. The entry 'client hw address' in the data field contains the MAC address of the wireless interface (00:16:6f:75:86:1e). The answer from the DHCP server is sent to this MAC address and broadcasted by the switches as unknown unicast, so it reaches the ethernet interface. Besides, a trace made on the wireless interface shows that no frame other than for 802.1X ever leaves the notebook on this interface and therefore the network never gets a chance to learn the MAC address of this interface.


Thanks,


Peter

Peter Paluch Mon, 09/13/2010 - 07:13
User Badges:
  • Cisco Employee,

Hi Peter,


Thank you for updating this thread! Interesting... you have seen the issue happening with DHCP packets. The MAC of the wireless interface was used in the hwaddr field of the DHCPREQ packet that was sent out the wired interface. Seems to be very consistent with the ARP issue I have described earlier. It could be interesting to see whether the ARPs behave in the same way but the DHCP does indeed.


Thank you again, Peter. This is really an issue worth having bookmarked somewhere.


Best regards,

Peter

vinod.agrahari Wed, 08/18/2010 - 06:45
User Badges:

Great,

I am not getting any reason to not get mac address through the command(sh mac).

Very interesting discussion.

Please just apply authentication to connect with AP and probably then you could get mac address from sh mac command.

Regards

Vinod Agrahari

Actions

This Discussion

Related Content