I just recovered from a strange issue that I am trying to figure out.
I get a call that users at a site cannot connect to only certain apps through my gre tunnel. I have an 871 on one side, and a 7206VXR on the other. I am using EIGRP.
In the midst of the problem, I could connect to most IP's in the 172.28.0.0/16 range from the site with the problem, but I could not connect to certain IP's. I looked at the route table, and all looked fine. I looked at the CEF table on the 871, and found a problem:
172.20.0.0/16 192.168.221.5 Tunnel0
172.21.0.0/16 192.168.221.5 Tunnel0
172.22.0.0/16 192.168.221.5 Tunnel0
172.23.0.0/16 192.168.221.5 Tunnel0
172.24.0.0/16 192.168.221.5 Tunnel0
172.26.0.0/16 192.168.221.5 Tunnel0
172.27.0.0/16 192.168.221.5 Tunnel0
172.28.0.0/16 192.168.221.5 Tunnel0
172.28.1.7/32 172.28.1.7 FastEthernet4
172.28.1.102/32 172.28.1.102 FastEthernet4
172.28.1.105/32 172.28.1.105 FastEthernet4
Note the few addresses in the 172.28.0.0/16 subnet being switched to fa4 instead of tunn0.
I then looked at the arp table, and found the same:
Internet 172.28.1.7 57 000e.d70b.d800 ARPA FastEthernet4
I look in the buffer, and see a couple of problems:
May 31 14:57:51: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 62: Neighbor 192.168.221.5 (Tunnel0) is down: holding time expired
May 31 14:58:13: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet4, changed state to down
May 31 14:58:17: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel0, changed state to down
May 31 14:58:24: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet4, changed state to up
May 31 14:58:27: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel0, changed state to up
May 31 14:58:31: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 62: Neighbor 192.168.221.5 (Tunnel0) is up: new adjacency
May 31 15:13:56: %SYS-5-CONFIG_I: Configured from console by jheckart on vty0 (172.28.1.211)
May 31 15:25:28: %IP_VFR-4-FRAG_TABLE_OVERFLOW: FastEthernet4: the fragment table has reached its maximum threshold 16
May 31 15:25:29: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 172.28.1.213 started - CLI initiated
May 31 15:44:54: %IP_VFR-4-FRAG_TABLE_OVERFLOW: FastEthernet4: the fragment table has reached its maximum threshold 16
May 31 15:57:04: %IP_VFR-4-FRAG_TABLE_OVERFLOW: FastEthernet4: the fragment table has reached its maximum threshold 16
I have an Ethernet handoff from the ISP. I have temporarily disabled virtual-reassembly, not knowing if it contributed to the problem. The fix, however, was to clear the arp table of the entries for the individual addresses within 172.28.0.0/16.
What happened here? I've used plenty of tunnels, and have not seen this problem before from a flapping connection.
If you do have a static (or floating static that is temporarily used) that point to an interface, then the router must ARP for any destination address that it will reach through that interface. And the behavior of IOS about the ARP table is that when it has a particular address in the ARP table it has a timer to expire the address. But as it expires the address it will issue a new ARP request and if it receives a response it will put the entry back into the table. So these ARP entries are likely to last for a very LONG time. It would take just a little bit of EIGRP converging to initiate this problem. Is it possible to re-write the static route to use a next hop address rather than using the interface? This would resolve the problem nicely.