Cisco 2811 router responds to arp requests for Virtual IP when in VRRP Backup mode

[sbaker000]

I have a Cisco 2811 series router running c2800nm-spservicesk9-mz.124-15.T10, a Cisco 3750 Switch running IOS Version 12.2(37)SE1, and a linux device.

The linux device is running as VRRP Master and the 2811 as Backup.  VRRP works normally in most conditions and transitions from Master to Backup as expected.  However, we have a problem when there is a layer 3 connectivity break between the 2811 and Linux device.

When the connection breaks, both the linux device and 2811 become Master since they are no longer exchanging VRRP Multicast messages with each other.  The 2811 sends a gratuitous arp to the switch informing it that the virtual ip now belongs to the Virtual MAC address of the router.  While this his happening, the Linux device retains master status since nothing changed that would cause it to fail to backup.

Once the Layer 3 connection issue is resolved, the switch continues to use the virtual MAC address in its arp table for the Virtual IP address and continues to respond to arp requests for the Virtual IP even though it has transitioned to backup.  The linux device also is responding to arp requests for the virtual ip however, the router is sending arp defend packets and the switch is retaining the virtual mac address in its arp table.

We have tried lowering the arp cache timeout to 30 seconds on the switch did not resolve the issue.  The only way we have found to resolve the issue is by clearing the arp cache on the Switch.

Is there any specific configuration on the switch or router that would preven this from happening?

[Jon Marshall]

Scott

Is the linux device connected directly into the 3750 as is the 2811 ?

It is also confusing when you talk about a loss of a L3 connection. VRRP, like HSRP and GLBP works over layer 2 not L3 so i'm not sure entirely what you mean. Can you clarify ?

Jon

[sbaker000]

We have seen this issue in 2 scenarios.  The first is with the linux device and router both directly connected to the Switch.  The second involves a Sonicwall firewall(in bridgemode) in between the linux device and switch and then also a Fluke between the Router and switch.  Both display the same symptoms.

  Sorry for the confusion, it was incorrect to say a loss of Layer 3 connection.  The problem only occurs when there is a break in connection between the Linux device and router in which the router and linux device are both VRRP Master at the same time (very brief period of time).  I expect to see some arp defend packets and arp issues on the switch while this is happening, but the problem continues after the connection between the 2 devices is restored.

[Jon Marshall]

Scott

I know it sounds a bit obvious but have you tried debugging vrrp on the 2811 when this happens to see what it does when it receives VRRP multicast packets from the Linux server after the link has been restored ?

Jon

[sbaker000]

Yes, the router is receiving the VRRP multicast packets from the linux device and successfully transitions back to VRRP backup.  The problem is that it continues to respond to arp requests for the virtual ip and also issues arp defend packets when it see's the linux device also responding to the arp requests.  Correct me if I am wrong here, but shouldn't the router stop responding to arp requests for the virtual ip once it has transitioned to backup?

[Jon Marshall]

sbaker000 wrote:
Yes, the router is receiving the VRRP multicast packets from the linux device and successfully transitions back to VRRP backup.  The problem is that it continues to respond to arp requests for the virtual ip and also issues arp defend packets when it see's the linux device also responding to the arp requests.  Correct me if I am wrong here, but shouldn't the router stop responding to arp requests for the virtual ip once it has transitioned to backup?

Yes it should. Once it becomes backup it should not be trying to answer any arp requests for the virtual IP.  Can you post config of switch and router if possible ?

Jon

[sbaker000]

I would have to scrub configs significantly before sharing.  The interface config for the router and switch are below:

2811

interface FastEthernet0/0
ip address 10.2.40.11 255.255.255.0
duplex full
speed 100
vrrp 1 ip 10.2.40.5
vrrp 1 priority 110
vrrp 1 track 1 decrement 15

3750

port connected to router

interface FastEthernet2/0/24
switchport mode access
speed 100
duplex full
snmp trap mac-notification added
snmp trap mac-notification removed
no mdix auto

port connected to linux

interface FastEthernet1/0/39
switchport mode access
speed 100
duplex full
snmp trap mac-notification added
snmp trap mac-notification removed
no mdix auto

I know there is probably nothing useful in these configs as it is pretty basic.  Let me know if there is something more specific you want to see off these devices.

Thanks

[Jon Marshall]

Scott

Nothing specific, just wanted to see if anything stood out. I'll have a quick look at Bug check when i get a moment just in case there is something there for either device. I'm assuming it is the 2811 that is misbehaving rather than the 3750, does this seem the most likely to you ?

When you do the debug on vrrp on the 2811 the link is definitely up and stable ? ie. there could be no flapping of the link. Apart from seeing it transition back to backup does the debug show anything else at all ?

Sorry for all the questions but like you i am a little nonplussed with this.

Jon

[sbaker000]

That is my presumption as well, that the 2811 is misbehaving.  The link is definitely stable and I did not see anything else that stood out in the debug logs.  In normal internet failures (as opposed to the LAN failure described) where the internet link to the Linux device fails everything works as we would expected and fails over to the router when the internet is down and then fails back to the linux device when it is restored.  The problem only arises when there is a break in communication on the LAN side.

[sbaker000]

I appreciate the help Jon. I am looking for any possible bugs related to this as well.  If you find something before I do, please let me know.

Thanks again

[Jon Marshall]

Scott

No problem. Likewise if you get to the bottom of it please let me know as well as it's a bit puzzling to say the least.

Jon