Nexus 5K with HSRP and Peer-Gateway

Answered Question
Mar 28th, 2012

I had an issue Live Migrating VMs and it boiled down to the fact that my pair of Nexus 5548's did not have their ARP tables sync'd.  I was told by TAC to remove the peer-gateway command from the VPC Domain.

vpc domain 1

role priority 100

peer-keepalive destination 10.8.128.2 source 10.8.128.1

delay restore 120

peer-gateway   <----- remove this

auto-recovery

Apparently there is an internal bug CSCto89187 in NX-OS version 5.0(3)N2(2) and the workaround is to remove the peer-gateway command which from my understanding this command simply routes packs destined to its peer MAC instead of sending it across the peer-link for the peer to route.

My question is this. Since I have HSRP configured, wouldn't the MAC address for packets be the Virtual MAC address which  is shared between both N5K's?  If this is true packets should always be destined to the same MAC, the virtual MAC, so how did removing the peer-gateway command fix my problem?

Thanks in advance!

I have this problem too.
0 votes
Correct Answer by brandonp@ca.ibm.com about 2 years 2 weeks ago

the peer-gateway command does like you stated: it allows the active router to forward packets it receives that are destined for the MAC address of the physical interface of its HSRP peer.

In theory, yes, everything should have a destination MAC address of the shared HSRP MAC address. However, there are some appliances and devices that don't follow the rules. Instead of referring to their ARP cache or forwarding tables when they reply to a packet, they have a feature that I've seen with various names, one of which is "mac reflection", where the device simply swaps the source and destination mac address when crafting its reply frame.

The HSRP MAC address is only used as a destination MAC address for packets destined to leave the network.

For packets coming ingress to the network, passing through the HSRP gateway, those frames will contain a source MAC of the physical interface of the HSRP gateway.

  • 1
  • 2
  • 3
  • 4
  • 5
Average Rating: 4.8 (6 ratings)
TervisTumbler Wed, 03/28/2012 - 12:45

So apparently it did not fix my issue.  The problem is not as bad now but it's hit or miss. I can communicate with other VLANs after migragation but only some machines.  I'm guessing it all depends on which way the traffic decides to go.

Looks like I'm back to square 1.  Any ideas?

TervisTumbler Wed, 03/28/2012 - 12:53

Great question!  That is was TAC recommened as well.  The problem is that statement is not available in my NX-OS version on the 5548UP.  If I upgrade my OS then I can try that and I have a strong feeling that is what my next step is going to be.

Ruisu.Cisco Thu, 03/29/2012 - 06:32

After a conversation with TAC, I was informed of the following. The ip arp synchronize statement should only sync ARP tables after a reboot of one of the N5Ks.  It does not continually sync the ARP tables.  I'm going to do a packet capture and see if the Hypervisor host is sending a gratitous ARP.  The IP addresses or MAC addresses are not changing so for the ARP table to be updated with an incorrect ARP entry may come from a gratitious ARP.

Ruisu.Cisco Thu, 03/29/2012 - 14:12

After some testing I notice it was only Nexus 1 that was updating it's ARP table incorrectly. Nexus 1 is the HSRP secondary but I'm not sure if that really means anything.

I captured packets on Nexus 1 during a live migration using the following command...

ethanalyzer local interface inbound-lo display-filter arp limit-capture-frames 10000 write bootflash:N5K1_ARP.pcp

I saw the VM that was moved perform a gratuitous arp with the correct MAC and IP.  Then, 6 packets later I saw the HyperVisor host perform three gratuitous arps in succession with it's MAC and the VMs IP address!!

Has anyone seen this before?  And by the way I'm using Hyper-V not VMware.

Correct Answer
brandonp@ca.ibm.com Fri, 03/30/2012 - 11:55

the peer-gateway command does like you stated: it allows the active router to forward packets it receives that are destined for the MAC address of the physical interface of its HSRP peer.

In theory, yes, everything should have a destination MAC address of the shared HSRP MAC address. However, there are some appliances and devices that don't follow the rules. Instead of referring to their ARP cache or forwarding tables when they reply to a packet, they have a feature that I've seen with various names, one of which is "mac reflection", where the device simply swaps the source and destination mac address when crafting its reply frame.

The HSRP MAC address is only used as a destination MAC address for packets destined to leave the network.

For packets coming ingress to the network, passing through the HSRP gateway, those frames will contain a source MAC of the physical interface of the HSRP gateway.

siddhartham Mon, 04/02/2012 - 09:37

F5 load balancers with auto last hop feature and EMC Celera are some of the devices that sends replys back to the source mac address instead of the gateway MAC address (Virtual HSRP MAC)

TervisTumbler Wed, 04/04/2012 - 06:36

I have to mark Brandon Premo response as the correct answer. I did packet captures and noticed the Nexus relaying packets and using its physical hardware address and the source address.

And I have to give thumbs up siddhartham for providing examples of products the flip source and destination addresses.

However, I am still having problems with this issue but I think it might be a setting in the NIC driver that is directly connected to my Nexus switches.  If anyone is interested in following up I have started another post here....  http://serverfault.com/questions/376433/nic-teaming-with-broadcom-on-hyperv-2008-r2

Actions

Login or Register to take actions

This Discussion

Posted March 28, 2012 at 7:09 AM
Stats:
Replies:8 Avg. Rating:4.75
Views:3761 Votes:0
Shares:0
Tags: hsrp, arp, nexus
+

Related Content

Discussions Leaderboard