Issue with OSPF Point-to-Multipoint over CES Cloud

[terrygwazdosky]

I'm having an issue running ospf point-to-multipoint over a CES cloud.  The reason I want to do this is that not every site has the same bandwidth and this way I can use neighbor statements to specify the cost to each neighbor.

To make this work I have to shut down the cloud facing interface on each node and bring them up one at a time.  Everything runs fine until I then shut one of them down and bring it up again, I'm only able to form adjanceis with 2-3 nodes - the other nodes get stuck in either INIT or EXSTART until OSPF give up do to too many retires.  If I then repeat the process of shutting down the interface on each node and bringing them back up it works again.

I've tried both " ip ospf network point-to-multipoint" and " ip ospf network point-to-multipoint non-broadcast" with the same results.

Here are the router types involved and the firmware they are running:

1. asr1006 asr1000rp1-adventerprisek9.03.10.00.S.153-3.S-ext.bin
2. asr1006 asr1000rp1-adventerprisek9.03.10.00.S.153-3.S-ext.bin
3. 2821 c2800nm-ipbasek9-mz.151-3.T4.bin
4. 2821 c2800nm-ipbasek9-mz.151-3.T4.bin
5. 2921 c2900-universalk9-mz.SPA.150-1.M1.bin (ipbasek9 license)
6. 2921  c2900-universalk9-mz.SPA.153-2.T.bin (ipbasek9 license)
7. 2821 c2800nm-ipbasek9-mz.151-3.T4.bin

Here is the relevant config from one of the routers:

interface GigabitEthernet1/0/6
 description CES
 bandwidth 50000
 ip address 10.226.126.30 255.255.255.224
 no ip redirects
 ip flow ingress
 ip flow egress
 ip ospf authentication message-digest
 ip ospf message-digest-key 1 md5 blahblahblah
 ip ospf network point-to-multipoint non-broadcast
 ip ospf dead-interval 3
 ip ospf hello-interval 1
 load-interval 30

!

router ospf 1
router-id 10.226.1.9
ispf
auto-cost reference-bandwidth 10000
timers throttle spf 10 100 5000
timers throttle lsa 10 100 5000
timers lsa arrival 80
passive-interface default  
no passive-interface GigabitEthernet1/0/6
network 10.226.126.0 0.0.0.31 area 0
neighbor 10.226.126.6 cost 1000
neighbor 10.226.126.5 cost 3333
neighbor 10.226.126.4 cost 3333
neighbor 10.226.126.3 cost 3333
neighbor 10.226.126.2 cost 3333
neighbor 10.226.126.1 cost 200

All the routers have at least one other interface running OSPF point-topoint with no issues.  The ASRs also have some stub areas in addition to area 0.  I've tried taking the ASRs out of the loop and testing, but the results are the same.

Please let me know if you have any ideas or need more details.

Thanks.

[Rolf Fischer · Accepted solution]

Hi,

this could be caused by proxy-ARP, which is enabled by default on your OSPF-interfaces.

Check the ARP-entries for the neighbors that don't form an adjacency and if necessary, do a "clear ip arp <neighbor-address>".

If the adjacencies come up then, disable proxy-ARP on the relevant interfaces.

HTH

Rolf
 

[terrygwazdosky]

I did a few preliminary tests and it appears you are correct.  It's funny because I always turn proxy-arp off on LAN interfaces.  It never occured to me that it could cause an issue on WAN interfaces.

Thank you so much, you really made my day.  :)

[Rolf Fischer]

You're welcome! Thanks for using the rating system!

Btw: A couple of months ago we had the very same problem with a similar OSPF setup over VPLS and it took us some time to find the cause ;-)

[Peter Paluch]

Hi Rolf,

Thanks for sharing the solution. I am confused, though: how can the ProxyARP be at the cause of this problem? ProxyARP does not respond to ARP requests if the target IP address lies in the IP subnet of the interface receiving the ARP request, and the OSPF communication here is only among routers in the same subnet - so deactivating the ProxyARP should not have any effect.

Can you perhaps provide a step-by-step scenario that explains how the ProxyARP impacts the connectivity? I am perplexed :-\

Best regards,
Peter

[Rolf Fischer]

Hi Peter,

believe me: I was perplexed as well and it took me some time to understand what was going on.

I've created a simple gns3 lab (topology attached) with handy IP- and MAC addresses:

R1: 192.168.0.1; 02:00:00:00:11:11
R3: 192.168.0.3; 02:00:00:00:33:33
R4: 192.168.0.4; 02:00:00:00:44:44

The host routes for the point-to-multipoint interfaces play an important role:

R3#show ip route 192.168.0.0
C       192.168.0.0/24 is directly connected, FastEthernet1/0
O       192.168.0.1/32 [110/1] via 192.168.0.1, 00:02:44, FastEthernet1/0
O       192.168.0.4/32 [110/64] via 172.16.34.4, 00:02:44, Serial0/1

Now, with a debug arp enabled on R1 we can see what happens when we shutdown and re-enable Fa1/0.

R1(config-if)#do show ip int brief f1/0
Interface                  IP-Address      OK? Method Status                Protocol
FastEthernet1/0            192.168.0.1     YES NVRAM  administratively down down
!
R1(config-if)#do show ip arp
<no output>
!

R1(config-if)#no shut
IP ARP: sent rep src 192.168.0.1 0200.0000.1111,
                 dst 192.168.0.1 ffff.ffff.ffff FastEthernet1/0
IP ARP: sent rep src 192.168.0.1 0200.0000.1111,
                 dst 192.168.0.1 ffff.ffff.ffff FastEthernet1/0
IP ARP: creating incomplete entry for IP address: 192.168.0.4 interface FastEthernet1/0
IP ARP: sent req src 192.168.0.1 0200.0000.1111,
                 dst 192.168.0.4 0000.0000.0000 FastEthernet1/0
IP ARP: creating incomplete entry for IP address: 192.168.0.3 interface FastEthernet1/0
IP ARP: sent req src 192.168.0.1 0200.0000.1111,
                 dst 192.168.0.3 0000.0000.0000 FastEthernet1/0
IP ARP: rcvd rep src 192.168.0.4 0200.0000.4444, dst 192.168.0.1 FastEthernet1/0
IP ARP: rcvd rep src 192.168.0.3 0200.0000.4444, dst 192.168.0.1 FastEthernet1/0
IP ARP: rcvd rep src 192.168.0.4 0200.0000.3333, dst 192.168.0.1 FastEthernet1/0
IP ARP: rcvd rep src 192.168.0.3 0200.0000.3333, dst 192.168.0.1 FastEthernet1/0
R1(config-if)#do show ip arp
Protocol  Address          Age (min)  Hardware Addr   Type   Interface
Internet  192.168.0.1             -   0200.0000.1111  ARPA   FastEthernet1/0
Internet  192.168.0.3             0   0200.0000.3333  ARPA   FastEthernet1/0
Internet  192.168.0.4             0   0200.0000.3333  ARPA   FastEthernet1/0

R1(config-if)#do show ip ospf neighbor
Neighbor ID     Pri   State           Dead Time   Address         Interface
4.4.4.4           0   INIT/  -        00:00:16    192.168.0.4     FastEthernet1/0
3.3.3.3           0   FULL/  -        00:00:16    192.168.0.3     FastEthernet1/0

OSPF: Rcv hello from 4.4.4.4 area 0 from FastEthernet1/0 192.168.0.4
OSPF: Send immediate hello to nbr 4.4.4.4, src address 192.168.0.4, on FastEthernet1/0
OSPF: Send hello to 192.168.0.4 area 0 on FastEthernet1/0 from 192.168.0.1
OSPF: End of hello processing

Because of the hostroute to R4, R3's proxy ARP answeres an ARP request for R4 (the same happens on R4)! At this point we have IP connectivity to R4 (via R3) but this doesn't work for OSPF's link local traffic.

So just for the fun of it, we could configure a static ARP entry to verify if that will fix the problem:

R1(config)#arp 192.168.0.4 0200.0000.4444 arpa
%OSPF-5-ADJCHG: Process 1, Nbr 4.4.4.4 on FastEthernet1/0 from LOADING to FULL, Loading Done

R1(config)#no arp 192.168.0.4 0200.0000.4444 arpa
IP ARP: creating incomplete entry for IP address: 192.168.0.4 interface FastEthernet1/0
IP ARP: sent req src 192.168.0.1 0200.0000.1111,
                 dst 192.168.0.4 0000.0000.0000 FastEthernet1/0
IP ARP: rcvd rep src 192.168.0.4 0200.0000.4444, dst 192.168.0.1 FastEthernet1/0
IP ARP: rcvd rep src 192.168.0.4 0200.0000.3333, dst 192.168.0.1 FastEthernet1/0

Or we could avoid that R3 installs an OSPF hostroute to 192.168.0.4.

R3(config)#ip prefix-list NO-HOSTROUTES deny 192.168.0.0/24 ge 32
R3(config)#ip prefix-list NO-HOSTROUTES permit 0.0.0.0/0 le 32
R3(config)#router ospf 1
R3(config-router)#distribute-list prefix NO-HOSTROUTES in
R3(config-router)#do show ip route 192.168.0.4
Routing entry for 192.168.0.0/24
  Known via "connected", distance 0, metric 0 (connected, via interface)

R1(config)#do clear ip arp 192.168.0.4
IP ARP: sent req src 192.168.0.1 0200.0000.1111,
                 dst 192.168.0.4 0200.0000.3333 FastEthernet1/0
IP ARP: creating incomplete entry for IP address: 192.168.0.4 interface FastEthernet1/0
IP ARP: sent req src 192.168.0.1 0200.0000.1111,
                 dst 192.168.0.4 0000.0000.0000 FastEthernet1/0
IP ARP: rcvd rep src 192.168.0.4 0200.0000.4444, dst 192.168.0.1 FastEthernet1/0
%OSPF-5-ADJCHG: Process 1, Nbr 4.4.4.4 on FastEthernet1/0 from LOADING to FULL, Loading Done

An interesting detail is the destination MAC address in the first request. Seems like the MAC address of the former ARP entry was still cached somewhere.

The adequate solution to solve the problem is of course disabling proxy-ARP.

I was quite surprised to see that that the protocol logic obviously ignores the fact of being in the same subnet but rather simply check if there is another entry for the requested host in the routing table.

The sanity checks section of RFC 1027 says

"An ARP subnet gateway implementation must not reply if the physical networks of the source and target of an ARP request are the same."

and it seems to me that IOS has not implemented this check. I'm looking forward to hear your opinion!

Thanks for joining,

best regards

Rolf

[Peter Paluch]

Rolf,

I have to replicate this is a lab and play with it a little as it is so utterly mind-boggling :) This is not to double check your claim - you are most probably spot on, it's just me needing to get my own feel about the issue.

I will be sure to respond ASAP.

Best regards,
Peter

[Rolf Fischer]

Hi Peter,

no problem at all, please take your time.

One more note: Instead of a hub (like the attached topology shows) I used a simple Switch to simulate a VPLS. Shouldn't make much of a difference, just for the sake of completeness ;-)

Best regards,

Rolf