%ADJ-5-PARENT: Midchain parent maintenance for IP midchain / EIGRP Problems

Unanswered Question
Apr 23rd, 2009
User Badges:

I have am using DMVPN which is working perfectly for domestic sites but as soon as I start using routers overseas I am getting the following error:


%ADJ-5-PARENT: Midchain parent maintenance for IP midchain out of Tunnel1, addr 10.48.12.2 854E1580 - looped chain attempting to stack



These routers are the exact same routers that we use in the United States and we are using the same config. I have no idea what's going on here.


Right now I have only one tunnel connecting to the routers and it's still complaining about the loop.


I've attached the config..


Thanks!!


Building configuration...



!

crypto isakmp policy 1

encr aes 256

hash md5

authentication pre-share

group 2

crypto isakmp key <removed> address 0.0.0.0 0.0.0.0 no-xauth

!

!

crypto ipsec transform-set AES esp-aes 256 esp-md5-hmac

mode transport

!

crypto ipsec profile AAA_VPN_2.0

set transform-set AES

!

!

bridge irb

!

!

interface Loopback0

ip address 10.100.1.52 255.255.255.255

!

interface Tunnel1

bandwidth 1500

ip address 10.48.12.22 255.255.254.0

no ip redirects

ip mtu 1200

ip nhrp authentication VPN20_T1

ip nhrp map 10.48.12.2 1.1.1.1

ip nhrp map 10.48.12.1 2.2.2.2

ip nhrp map multicast 1.1.1.1

ip nhrp map multicast 2.2.2.2

ip nhrp network-id 1

ip nhrp nhs 10.48.12.2

ip nhrp nhs 10.48.12.1

ip nhrp registration no-unique

tunnel source FastEthernet4

tunnel mode gre multipoint

tunnel protection ipsec profile AAA_VPN_2.0

!

interface FastEthernet0

!

interface FastEthernet1

!

interface FastEthernet2

!

interface FastEthernet3

!

interface FastEthernet4

ip address dhcp

ip nat outside

ip virtual-reassembly

duplex auto

speed auto

!

interface Dot11Radio0

no ip address

shutdown

speed basic-1.0 basic-2.0 basic-5.5 6.0 9.0 basic-11.0 12.0 18.0 24.0 36.0 48.0 54.0

station-role root

bridge-group 1

bridge-group 1 subscriber-loop-control

bridge-group 1 spanning-disabled

bridge-group 1 block-unknown-source

no bridge-group 1 source-learning

no bridge-group 1 unicast-flooding

!

interface Vlan1

no ip address

bridge-group 1

bridge-group 1 spanning-disabled

!

interface BVI1

ip address 10.32.50.1 255.255.255.0

ip nat inside

ip virtual-reassembly

!

router eigrp 10

passive-interface default

no passive-interface Tunnel1

network 10.48.12.22 0.0.0.0

network 10.100.1.52 0.0.0.0

no auto-summary

!


!

!

!

logging dmvpn

!

!

!

!

tacacs-server host 172.27.10.212 key 7 <removed>

tacacs-server host 10.2.100.208 key 7 <removed>

tacacs-server directed-request

!

control-plane

!

bridge 1 route ip

!

line con 0

no modem enable

line aux 0

line vty 0 4

transport input ssh

!

scheduler max-task-time 5000

end







  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 5 (3 ratings)
Loading.
Laurent Aubert Thu, 04/23/2009 - 10:34
User Badges:
  • Cisco Employee,

Hi,


Usually, this message is related to routing loop where you are learning tunnel destination addresses inside the tunnel.


Could you check which routes the router is using to join 1.1.1.1 and 2.2.2.2 ?


Thanks


Laurent.

okandaur89 Thu, 05/30/2013 - 00:09
User Badges:

Can this problame be the cause of 881 router crashing?

Using IOS 12.4(20).T3 and this log is last before crashing:


*May 19 08:17:56.909: %ADJ-5-PARENT: Midchain parent maintenance for IP midchain out of Tunnel1, addr 192.168.188.65 (incomplete) - looped chain attempting to stack
Richard Burts Thu, 05/30/2013 - 06:14
User Badges:
  • Super Silver, 17500 points or more
  • Hall of Fame,

    Founding Member

  • Cisco Designated VIP,

    2017 LAN, WAN

I would not think that this error would cause a crash of a router. I have seen that error many times when a GRE tunnel is configured and the physical interface that terminates the tunnel changes state to down. When you see the error message %ADJ-5-PARENT is there perhaps a message just before it about a physical interface changing state to down?


If the router crashes does it produce a crashinfo file? If so there would perhaps be information in the crashinfo file that might help understand the cause of the crash.


HTH


Rick

okandaur89 Thu, 05/30/2013 - 07:25
User Badges:

I took this log exectly from the crashinfo file. It is very big for posting here, and this log string was last recorded before crashing.

Richard Burts Thu, 05/30/2013 - 12:35
User Badges:
  • Super Silver, 17500 points or more
  • Hall of Fame,

    Founding Member

  • Cisco Designated VIP,

    2017 LAN, WAN

Crashinfo files are frequently pretty big. And without seeing what is in the crashinfo file it may be difficult to know for sure what caused the crash. Sometimes the output of show version may have some information about what caused the last restart. Perhaps you could post the output of show version for your 881?


Is this router under a maintenance contract? If so then opening a case with Cisco TAC is a possibility and they may be able to determine the cause of the crash.


I can only say that I have seen this log message quite a bit and I have yet to see it cause a crash on the routers that I was working on.


HTH


Rick

Richard Burts Fri, 05/31/2013 - 18:37
User Badges:
  • Super Silver, 17500 points or more
  • Hall of Fame,

    Founding Member

  • Cisco Designated VIP,

    2017 LAN, WAN

Thanks for posting the crashinfo file. I have taken a look at the file. Unfortunately I do not have the proper tools to fully interpret the crashinfo. A full interpretation would require the resources available through Cisco TAC. The crashinfo does allow us to make some observations and suggestions. My primary observation is that the crashinfo file has multiple references to traceback such as this one

-Traceback= 0x8039D95C 0x8039D8B8 0x800A4320 0x800A52C4 0x800A5B80 0x8039F9AC 0x803A273C


I do not have access to tools that would interpret this particular traceback but I can observe that traceback is almost always an indication of a software problem. And the usual solution to a problem involving traceback is to change the version of code that the router is running.


HTH


Rick

Mark Mattix Wed, 03/04/2015 - 10:50
User Badges:

Hello Richard, I'm also using a DMVPN configuration and just had this issue in my network. Before I saw:
%ADJ-5-PARENT: Midchain parent maintenance for IP midchain out of Tunnel0 - looped chain attempting to stack
I saw the physical interface protocol go down. This actually happened to 2 sites that were within 10 miles of each other. Do you think it was an ISP issue that made the physical interfaces go down or a problem related to OSPF? I just implemented OSPF network wide this past Monday so it has me concerned. Thanks a lot for your advice!  -Mark

Richard Burts Wed, 03/04/2015 - 11:09
User Badges:
  • Super Silver, 17500 points or more
  • Hall of Fame,

    Founding Member

  • Cisco Designated VIP,

    2017 LAN, WAN

Mark

 

I am fairly certain that this reflects a physical/ISP problem and not an issue with OSPF. Your observation that you saw a message about the physical interface protocol go down pretty much confirms this. I do not see how OSPF could make a physical interface protocol go down.

 

Keep an eye on it and let us know what you find. But I am confident that it is not an OSPF issue.

 

HTH

 

Rick


Mark Mattix Wed, 03/04/2015 - 11:12
User Badges:

Thank you Richard, I'll let you know if I experience this again.

tbonfigli Mon, 02/01/2016 - 07:32
User Badges:

Richard/Mark:

Hello and good morning!  I just wanted to add to what Richard is suggesting (that it is a physical/ISP issue and not an OSPF/recursion issue) with respect to what the probable cause could be.  We are running a point-to-point SVTI between Maryland and Alaska and everything has been working great for about 2 months now.  Then, all of sudden, yesterday I get a SolarWinds alert indicating that my tunnel interfaces on either end had gone down.  Here is the log output from both routers (full disclosure, I changed the tunnel/IP addresses from the originals):

Maryland Router:

000236: *Jan 31 2016 01:11:35.793 UTC: %ADJ-5-PARENT: Midchain parent maintenance for IP midchain out of Tunnel64620 - looped chain attempting to stack
000237: *Jan 31 2016 01:11:42.365 UTC: %TUN-5-RECURDOWN: Tunnel64620 temporarily disabled due to recursive routing
000238: *Jan 31 2016 01:11:42.365 UTC: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel64620, changed state to down
000239: *Jan 31 2016 01:11:42.365 UTC: %OSPF-5-ADJCHG: Process 64620, Nbr 100.200.130.254 on Tunnel64620 from FULL to DOWN, Neighbor Down: Interface down or detached
000240: *Jan 31 2016 01:16:32.825 UTC: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel64620, changed state to up
000241: *Jan 31 2016 01:16:41.285 UTC: %OSPF-5-ADJCHG: Process 64620, Nbr 100.200.130.254 on Tunnel64620 from LOADING to FULL, Loading Done

...and here is the Alaska router:

00282: Jan 31 2016 01:12:09.930 UTC: %ADJ-5-PARENT: Midchain parent maintenance for IP midchain out of Tunnel64621 - looped chain attempting to stack
000283: Jan 31 2016 01:12:13.418 UTC: %TUN-5-RECURDOWN: Tunnel64621 temporarily disabled due to recursive routing
000284: Jan 31 2016 01:12:13.418 UTC: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel64621, changed state to down
000285: Jan 31 2016 01:12:13.418 UTC: %OSPF-5-ADJCHG: Process 64621, Nbr 100.200.100.253 on Tunnel64621 from FULL to DOWN, Neighbor Down: Interface down or detached
000286: Jan 31 2016 01:16:53.891 UTC: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel64621, changed state to up
000287: Jan 31 2016 01:16:54.335 UTC: %OSPF-5-ADJCHG: Process 64621, Nbr 100.200.100.253 on Tunnel64621 from LOADING to FULL, Loading Done

When I first saw these log messages, my immediate thoughts were that we couldn't have "recursive" routing just all-of-a-sudden pop up out of nowhere.  If there was recursive routing we should have/would have seen it long before this incident happened on the 31st.  A quick search of the support forums (always my "go to" when things like this pop up!) turned up this thread and since I haven't seen this error before I was thrilled to find Richard's feedback that it is either a physical problem or something with the ISP.

Ironically, about 3 months ago we transitioned the connectivity between these two routers to a 3rd party provider's MPLS L3 VPN network.  I fired off a copy of our logs to see if they might have seen anything during this time frame and "BINGO"!  They responded by letting me know that they had performed a JUNOS upgrade on their router which links from Seattle to Alaska...nailed it! :-)  Now we just need to figure out why they never told us they were doing that! ;-)))

So, to Richard's point, it definitely appears that you can end up with this error if there is something going on in the ISPs network that causes a loss in connectivity.

Also wanted to say, "THANK YOU" to Richard (whose posts I always seem to come across when looking for information!) for always sharing what seems to be "spot on" information!!!

Cheers,

Travis


Richard Burts Mon, 02/01/2016 - 11:15
User Badges:
  • Super Silver, 17500 points or more
  • Hall of Fame,

    Founding Member

  • Cisco Designated VIP,

    2017 LAN, WAN

Travis


Thank you for the kind words about my contributions. I am glad that you have found them helpful and hope that you will continue to be active and to post to the forum.


I remember the case that I opened with Cisco TAC when I encountered this issue and am surprised that I did not include an explanation of this symptom for this forum. So let me address that now.

We believe that this is what is happening:

- the router has a route for the tunnel endpoint that points to a next hop out some physical interface (this is pretty necessary to avoid the recursive routing problem with the tunnel).

- the router runs a dynamic routing protocol over the tunnel and learns routes (probably including a default route) through the tunnel.

- some event impacts the physical interface or something causes it to go to the protocol down state.

- the router withdraws the route to the tunnel end point (since the next hop is not available or the outbound interface is in the protocol down state).

- for a brief period the dynamic learned route is still present in the routing table and the router believes that it can reach the tunnel end point by going through the tunnel (which is the classic definition of recursive routing). So the router generates the midchain parent maintenance error message.

- the dynamic routing protocol converges and removes the route from the routing table.

- now things can return to normal, wait for the physical interface to recover, and re-establish the tunnel.


So the midchain parent maintenance error is essentially an issue of timing of how quickly the route based on the physical interface is removed vs how quickly the dynamic learned route is removed. And in my experience the midchain parent maintenance message is always related to an issue with the connection going out the outbound interface (or perhaps an issue in the provider network).


HTH


Rick 

Actions

This Discussion