mGRE tunnel SAs negotiated, but don't survive.

Answered Question
Jul 21st, 2010
User Badges:
  • Silver, 250 points or more

Background


We have a stable P2P GRE + IPSec configuration with multiple spokes using rsa-signatures for ISAKMP authentication, and EIGRP as the routing protocol. We are transitioning to an mGRE configuration (DMVPN). The P2P GRE tunnel interfaces are administratively shutdown, the crypto maps on the physical interfaces have been removed, and the crypto database has been cleared.



Issue


When we bring up the mGRE tunnel interfaces (hub and spoke), we are able to complete ISAKMP phase I and II (briefly). However, ~ 1-1/2 minutes later, we see a debug message on the hub, such as:


Jul 21 13:56:49.601 EDT: IPSEC(cleanup_tun_decap_oce): unlock and null out Tunnel0 tun_decap_oce 86742E48 from ident 86FB990C


... and then the IPSec SAs are deleted, the tunnel comes down, IKE_PHASE2_DEL and IKE_PHASE1_DEL messages are generated, and we start over with ISAKMP phase I negotiation.


Anyone know what the "oce" stands for?



Debug (ISAKMP & IPSec) Highlights


Jul 21 13:55:13.188 EDT: ISAKMP:(2597):SA authentication status: authenticated
Jul 21 13:55:13.236 EDT: ISAKMP:(2597):Old State = IKE_R_MM5  New State = IKE_P1_COMPLETE
Jul 21 13:55:13.356 EDT: IPSEC(create_sa): sa created,
Jul 21 13:55:13.356 EDT: IPSEC(create_sa): sa created,
Jul 21 13:55:13.356 EDT: %CRYPTO-5-SESSION_STATUS: Crypto tunnel is UP  .  Peer <spoke-physical-ip>:500       Id: spoke.domain.null
Jul 21 13:55:13.356 EDT: %DMVPN-7-CRYPTO_SS: Tunnel0-<hub-physical-ip> socket is UP
Jul 21 13:55:13.700 EDT: ISAKMP:(2597):Old State = IKE_QM_R_QM2  New State = IKE_QM_PHASE2_COMPLETE
Jul 21 13:56:49.601 EDT: IPSEC(cleanup_tun_decap_oce): unlock and null out Tunnel0 tun_decap_oce 86742E48 from ident 86FB990C
Jul 21 13:56:49.601 EDT: IPSEC(delete_sa): deleting SA,
Jul 21 13:56:49.601 EDT: IPSEC(delete_sa): deleting SA,
Jul 21 13:56:49.601 EDT: %CRYPTO-5-SESSION_STATUS: Crypto tunnel is DOWN.  Peer <spoke-physical-ip>:500       Id: spoke.domain.null
Jul 21 13:56:49.601 EDT: ISAKMP:(2597):Input = IKE_MESG_FROM_IPSEC, IKE_PHASE2_DEL
Jul 21 13:56:49.605 EDT: ISAKMP:(2597):Input = IKE_MESG_INTERNAL, IKE_PHASE1_DEL


Note: A more complete debug output is attached.



General Observations (sh crypto isakmp sa, sh crypto ipsec sa)


ISAKMP SA reaches a state of QM_IDLE, and a status of ACTIVE. However, the SA is deleted and a new one is spawned within ~ minute.


IPSec SAs are negotiated on the hub and spoke. However, only the spoke has packet encaps, and only the hub has decaps. Wireshark confirms that the hub is not placing any ESP packets on the wire. The IPSec SAs are deleted, and new ones are spawned every ~1-1/2 minutes.



Show Command Output


hub#sh cry ipsec profile
IPSEC profile DMVPN
        Security association lifetime: 4608000 kilobytes/3600 seconds
        Responder-Only (Y/N): N
        PFS (Y/N): Y
        DH group:  group2
        Transform sets={eni-xfm-des:  { esp-des esp-sha-hmac  } , eni-xfm-3des:  { esp-3des esp-sha-hmac  } ,}


hub#sh cry map
Crypto Map "Tunnel0-head-0" 65536 ipsec-isakmp
        Profile name: DMVPN
        Security association lifetime: 4608000 kilobytes/3600 seconds
        Responder-Only (Y/N): N
        PFS (Y/N): Y
        DH group:  group2
        Transform sets={eni-xfm-des:  { esp-des esp-sha-hmac  } , eni-xfm-3des:  { esp-3des esp-sha-hmac  } ,}


Crypto Map "Tunnel0-head-0" 65537 ipsec-isakmp
        Map is a PROFILE INSTANCE.
        Peer = <spoke-physical-ip>
        Extended IP access list
            access-list  permit gre host <hub-physical-ip> host <spoke-physical-ip>
        Current peer: <spoke-physical-ip>
        Security association lifetime: 4608000 kilobytes/3600 seconds
        Responder-Only (Y/N): N
        PFS (Y/N): Y
        DH group:  group2
        Transform sets={eni-xfm-des:  { esp-des esp-sha-hmac  } , eni-xfm-3des:  { esp-3des esp-sha-hmac  } ,}
        Interfaces using crypto map Tunnel0-head-0:Tunnel0


hq-edg01#sh cry session detail
Crypto session current status


Interface: Tunnel0
Uptime: 00:00:10
Session status: UP-ACTIVE
Peer: <spoke-physical-ip> port 500 fvrf: (none) ivrf: (none)
      Phase1_id: spoke.domain.null
      Desc: (none)
  IKE SA: local <hub-physical-ip>/500 remote <spoke-physical-ip>/500 Active
          Capabilities:(none) connid:2682 lifetime:23:59:47
  IKE SA: local <hub-physical-ip>/500 remote <spoke-physical-ip>/500 Inactive
          Capabilities:(none) connid:2681 lifetime:0
  IPSEC FLOW: permit 47 host <hub-physical-ip> host <spoke-physical-ip>
        Active SAs: 2, origin: crypto map
        Inbound:  #pkts dec'ed 6 drop 0 life (KB/Sec) 4517257/3589
        Outbound: #pkts enc'ed 0 drop 0 life (KB/Sec) 4517258/3589



Hardware & IOS


c1811 (hub) - c181x-advipservicesk9-mz.124-24.T
c1711 (spoke) - c1700-advipservicesk9-mz.124-15.T9



Relevant crypto portions of the DMVPN configurations (hub/spoke) follow:


crypto isakmp policy 3
encr 3des
group 2
lifetime 86399


crypto isakmp identity hostname


crypto ipsec transform-set eni-xfm-3des esp-3des esp-sha-hmac
mode transport
crypto ipsec transform-set eni-xfm-des esp-des esp-sha-hmac
mode transport


crypto ipsec profile DMVPN
set security-association lifetime seconds 3600
set transform-set eni-xfm-des eni-xfm-3des
set pfs group2


interface Tunnel0
ip address <removed> 255.255.255.0
tunnel protection ipsec profile DMVPN


Note: NHRP,  mGRE, and most other parameters have been snipped.


Any assistance would be appreciated.


Best Regards,
Mike

Correct Answer by Michael Sullenberger about 6 years 7 months ago

You are correct in your observation.


The previous p-pGRE interface (in your case) can get its information in to

the tunnel endpoint database (controls tunneling packets) even if the

p-pGRE tunnel is shutdown.  Also in the code a GRE packet that is

destined to the router will check for a mathc with a p-pGRE tunnel before

mGRE tunnels. So the inbound GRE tunnel packets were getting "grabbed"

by the p-pGRE tunnel and then dropped.


If I really want a GRE tunnel to be "down" I will remove the 'tunnel source ...'.

If I want to have both tunnels up at the same time I do what you did, give

each one a different tunnel key or a different tunnel source.


Hope this helps explain what was happening.


Mike.


PS. You should be able to mark this as answered now.

  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 5 (1 ratings)
Loading.
michael.leblanc Wed, 07/21/2010 - 22:56
User Badges:
  • Silver, 250 points or more

To All:


We've resolved the issue, and learned a lesson.


We were migrating from a P2P GRE to mGRE configuration. In doing so, we defined new tunnel interfaces for mGRE, and shutdown the P2P tunnel interfaces. We migrated configuration parameters we wished to retain, including the "tunnel key". We assumed erroneously, that we could do so given that the P2P tunnels were administratively shutdown. NOT!


On a hunch, we shutdown the mGRE tunnel interfaces, modified the tunnel key to a value different than that configured on the shutdown P2P tunnel interfaces, and then brought the mGRE interfaces out of shutdown. The tunnel came up, stayed up, NHRP registration took place, routes were exchanged, and we breathed a sigh of relief.


Best Regards,

Mike

michael.leblanc Wed, 07/21/2010 - 23:00
User Badges:
  • Silver, 250 points or more

To Anyone:


How do we mark our own post as "answered"?


Best Regards,


Mike

Correct Answer
Michael Sullenberger Tue, 08/10/2010 - 17:02
User Badges:
  • Cisco Employee,

You are correct in your observation.


The previous p-pGRE interface (in your case) can get its information in to

the tunnel endpoint database (controls tunneling packets) even if the

p-pGRE tunnel is shutdown.  Also in the code a GRE packet that is

destined to the router will check for a mathc with a p-pGRE tunnel before

mGRE tunnels. So the inbound GRE tunnel packets were getting "grabbed"

by the p-pGRE tunnel and then dropped.


If I really want a GRE tunnel to be "down" I will remove the 'tunnel source ...'.

If I want to have both tunnels up at the same time I do what you did, give

each one a different tunnel key or a different tunnel source.


Hope this helps explain what was happening.


Mike.


PS. You should be able to mark this as answered now.

michael.leblanc Wed, 08/11/2010 - 07:47
User Badges:
  • Silver, 250 points or more

Michael:


Thank you for the additional background info.


We'll make a point of removing the tunnel source when retaining a shutdown tunnel interface in future configs.


Best Regards,


Mike.

Actions

This Discussion

Related Content