ARP table aging, asymmetric routing, and unicast flooding

TYLER WEST · ‎02-14-2006

We have built a couple of high-availability environments with 6500s and Sup720s. We have been concerned with the issue of asymmetric routing and unicast flooding and have observed this behavior on more than one occasion. My question stems from documents located at the following URLs:

http://www.cisco.com/en/US/customer/products/hw/switches/ps700/products_tech_note09186a00801d0808.shtml

http://www.cisco.com/en/US/customer/tech/tk648/tk362/technologies_tech_note09186a0080094afd.shtml#t8

The solution given is to bring the ARP timeout value and the mac-address aging value closer to each other with the preference being to increase the mac-address timer. However I am not sure this really solves the problem. If I set my mac addresses to age out after 4 hours then I think the problem will still occur, it will just be 3 hours and 55 minutes later than normal. This is especially true of 24/7 hosts such as servers that are going to be communicated with at least once every 4 hours.

The behavior I have observed is that the packets passing through a router destined for a host reset the ARP age to zero in the router. Therefore a host that is communicated with continuously will never age out its ARP entry but the mac-address in the CAM or mac-address-table will still age out because the switch with the MSFC operating as the standby HSRP gateway never sees a packet with the source mac address of the host. Therefore another ARP request is never sent by the standby MSFC so no ARP reply with the host's source mac address is sent allowing the update of the mac-address table on that switch.

Is there a way to force the ARP table to age its entries regardless of the traffic?

Tyler West

Sr. Network Engineer

Dollar General Corporation

TYLER WEST · ‎02-15-2006

And while anyone is looking at this, is there any sort of statistics or logging on the switches that could be used to identify unicast flooding and/or the destination mac-addresses that are causing the flooding?

Thanks again,

Tyler

jarathbu · ‎02-16-2006

Catalyst 6500/6000 Supervisor Engine 2 and higher series switches running Cisco IOS System software (Native) version 12.1(14)E and higher or Cisco CatOS system software version 7.5 or higher implements 'unicast flood protection' feature. This feature allows the switch to monitor the amount of unicast flooding per VLAN and take specified action if flooding exceeds specified amount. Actions can be to syslog, limit or shutdown VLAN - the syslog being the most useful for flood detection generating messages similar to:

%UNICAST_FLOOD-4-DETECTED: Host 0000.0000.2100 on vlan 1 is flooding

cat6000#sh mac-address-table unicast-flood

If your switch doesn’t support unicast flood protection you can set up a span port and monitor traffic for an impacted host. From the sniffer trace you should see multiple packets (flooded packets) stand out. These packets would be packets that are not sourced or destined to the address of the device on that port (excluding broadcast/multicast traffic of course).

jarathbu · ‎02-16-2006

Hello,

On a sup720 with "switchport" capable ports you can configure UUFB - unknown unicast flooding blocking. For additional information:

http://www.cisco.com/en/US/products/hw/switches/ps708/products_configuration_guide_chapter09186a0080435871.html

Hope this helps.

Regards,

James

Richard Burts · ‎02-17-2006

I find the responses about control of unicast flooding to be very interesting and helpful. But I would also question one of the bases of the original question. Tyler believes that if traffic is sent to an address (perhaps a server) that it continually resets the ARP entry. That has not been my experience. What I see is that the entry is created, put into the table, is aged, and is refreshed only when there is an ARP response from the device (the request might have come from the IOS device or from some other device within the broadcast domain).

If Tyler believes otherwise it would be interesting to run debug arp and then do a series of show arp outputs that show the entry being refreshed without any arp activity in the debug.

HTH

Rick

HTH

Rick

TYLER WEST · ‎02-17-2006

I can't speak for other Cisco L3 devices at this point but the experience I am having shows that the ARP table entry is having the age reset to zero for every packet that passes through that MSFC destined for the host in question. When the table doesn't have an entry we can send a ping to that host from another host on a different VLAN. The MSFC will send an ARP request to which the destination host will reply. This will put an entry in the ARP table with an age of zero. It will also generally update the mac-address-table along the path we are concerned with. I can monitor the age of the ARP entry and before it ages out (4 hours by default) send another ping packet. At that point I can show the entry in the ARP table and it has reset the age back to zero.

In an HSRP configuration, if the standby HSRP MSFC is the one receiving the packet that is destined for the host he will refresh his ARP table entry from the ARP reply sent from the host and send the packet on its merry way. However, when the host replies to the packet itself he will send that packet to the active HSRP MSFC on the other switch and not to the standby HSRP MSFC. If you continue with the ping packets then the ARP table entry on the standby HSRP MSFC continuously gets reset to zero but the mac-address-table entry will eventually age out because that switch is never seeing any return traffic from the host.

I would agree with you that it should only reset with an ARP reply and I think that would be in accordance with RFC standards. That is not the behavior we are seeing however. We're continuing in the lab to test and observe to make sure we completely understand the behavior we are seeing but so far that seems to be the case.

Thanks,

Tyler

TYLER WEST · ‎02-17-2006

But unfortunately that breaks the normal operation of unicast flooding to try and get the FIRST frame to its destination host in the hopes that a reply will refresh the mac-address-table. This feature requires the introduction of static mac-address entries for those ports.

Thanks,

Tyler

TYLER WEST · ‎02-17-2006

Thank you. It was actually monitoring the traffic on a host on that VLAN for which the traffic was not destined that revealed the incident. Please understand that, at this point, this is not very impacting but the potential for significant impact is there.

The unicast flood monitoring feature you describe is apparently only available on Sup2 and not on Sup720. I stumbled across that one as well and hoped it would help but it did not. Thanks for your time.

Tyler