I had an issue Live Migrating VMs and it boiled down to the fact that my pair of Nexus 5548's did not have their ARP tables sync'd. I was told by TAC to remove the peer-gateway command from the VPC Domain.
vpc domain 1
role priority 100
peer-keepalive destination 10.8.128.2 source 10.8.128.1
delay restore 120
peer-gateway <----- remove this
Apparently there is an internal bug CSCto89187 in NX-OS version 5.0(3)N2(2) and the workaround is to remove the peer-gateway command which from my understanding this command simply routes packs destined to its peer MAC instead of sending it across the peer-link for the peer to route.
My question is this. Since I have HSRP configured, wouldn't the MAC address for packets be the Virtual MAC address which is shared between both N5K's? If this is true packets should always be destined to the same MAC, the virtual MAC, so how did removing the peer-gateway command fix my problem?
Thanks in advance!
the peer-gateway command does like you stated: it allows the active router to forward packets it receives that are destined for the MAC address of the physical interface of its HSRP peer.
In theory, yes, everything should have a destination MAC address of the shared HSRP MAC address. However, there are some appliances and devices that don't follow the rules. Instead of referring to their ARP cache or forwarding tables when they reply to a packet, they have a feature that I've seen with various names, one of which is "mac reflection", where the device simply swaps the source and destination mac address when crafting its reply frame.
The HSRP MAC address is only used as a destination MAC address for packets destined to leave the network.
For packets coming ingress to the network, passing through the HSRP gateway, those frames will contain a source MAC of the physical interface of the HSRP gateway.