Interesting issue indeed. Does this happen on a hub or on a spoke router? Can you paste the complete configuration of your Tunnel interface here (both spoke and hub)? Also it would be interesting to see the output of the following debugs:
debug ip ospf hello
debug ip ospf adj
on both sides of the mGRE tunnel (take only two routers - hub and a selected spoke, some of them stuck in the INIT state).
An adjacency stuck in the INIT state means that our router is receiving and understanding the OSPF Hello packets from the neighbor but the neighbor is not seeing our router. Thus it looks like the mGRE tunnel becomes "unidirectional" for whatever reason (don't take me at the word right now, I'm just describing my first impression). That could have something to do with multicast mappings.
interface Tunnel1101 description DMVPN tunnel ip address 172.17.5.1 255.255.255.0 no ip redirects ip flow ingress ip nhrp authentication ***** ip nhrp map multicast dynamic ip nhrp network-id 1001 ip ospf network point-to-multipoint ip ospf cost 100 ip ospf hello-interval 10 ip ospf mtu-ignore load-interval 30 tunnel source x.x.x.x tunnel mode gre multipoint tunnel key 1101 end
interface Tunnel1101 description Multinet Link to HUB ip address 172.17.5.44 255.255.255.0 ip flow ingress ip nhrp authentication **** ip nhrp map 172.17.5.1 x.x.x.x ip nhrp map multicast x.x.x.x ip nhrp network-id 1001 ip nhrp nhs 172.17.5.1 ip nhrp registration timeout 35 ip ospf network point-to-point ip ospf cost 110 ip ospf mtu-ignore tunnel source a.b.c.d tunnel destination x.x.x.x tunnel key 1101
One thing i want to re-emphasize, that everything is working fine, usually it happens when router reloads but it has also occured while working as well. Now as soon as i delete the tunnel and recreate it with different tunnel number, adjcaencies come up !!!
I have noticed an interesting issue in your configuration and I'd like to discuss it with you. On your spoke router, you seem to be using a point-to-point GRE tunnel. Is that a purpose? With point-to-point GRE tunnel on your spoke, you cannot have a spoke-to-spoke tunnel and all inter-spoke traffic must go through the hub router, thereby defeating one of major advantages of the DMVPN solution.
Allow me please a few more questions:
Which router is seen in the INIT state? A hub or a spoke?
Are the NHRP mappings and the CEF table absolutely correct both on the hub and the spoke in the moment of the problem?
Have you tried clearing the NHRP mappings or the entire CEF table and see whether the problem is removed?
Will you be able to peform the debug commands I've suggested in my previous post?
I admit, this does look like a bug but I have a feeling that something is triggering it and I would like to see where exactly does this bug appear and whether it can manifest itself also in some show commands, not just by OSPF adjacencies falling.
We have seen similar issues in the past due to IOS bugs, one of the more recent one is this http://tools.cisco.com/Support/BugToolKit/search/getBugDetails.do?method=fetchBugDetails&bugId=CSCsv43385. Note even though in the bug it describes removing and re-adding the same tunnel interface can be used as a workaround, it's not guaranteed given the problem is a caused by an internal timing condition in the code. This bug does affect 12.4(20)T and 12.4(22)T, although it should be fixed in 12.4(22)T2. I would suggest you upgrade the code to at least get pass this bug. If you still see the same issue, you should check to make sure the CEF adjacency is consistent with the NHRP mappings, and you can do this by "show adj tunnelx internal" to make the adj encap is the same as that in "show ip nhrp". However, the adjacency rewrite information is in hex, so you'd have to look for it and it's probably a better idea to open a TAC case to address it at that point.
I really want to thank you guys for helping me out. I will change the IOS today and will update you tomorrow if it solves the issue. Also Peter, just in case if the issue persists, i will paste all the debugs here and will take your described actions to see where the issue lies. Regarding Spoke end P2P GRE, yes its our requirement since we are using centralized servers, and spokes doesnt need to cummunicate with any other spoke.
We are pleased to announce availability of Beta software for 16.6.3. 16.6.3 will be the second rebuild on the 16.6 release train targeted towards Catalyst 9500/9400/9300/3850/3650 switching platforms. We are looking for early feedback from custome...