Solved: mGRE tunnel SAs negotiated, but don't survive.

michael.leblanc · ‎07-21-2010

Background

We have a stable P2P GRE + IPSec configuration with multiple spokes using rsa-signatures for ISAKMP authentication, and EIGRP as the routing protocol. We are transitioning to an mGRE configuration (DMVPN). The P2P GRE tunnel interfaces are administratively shutdown, the crypto maps on the physical interfaces have been removed, and the crypto database has been cleared.

Issue

When we bring up the mGRE tunnel interfaces (hub and spoke), we are able to complete ISAKMP phase I and II (briefly). However, ~ 1-1/2 minutes later, we see a debug message on the hub, such as:

Jul 21 13:56:49.601 EDT: IPSEC(cleanup_tun_decap_oce): unlock and null out Tunnel0 tun_decap_oce 86742E48 from ident 86FB990C

... and then the IPSec SAs are deleted, the tunnel comes down, IKE_PHASE2_DEL and IKE_PHASE1_DEL messages are generated, and we start over with ISAKMP phase I negotiation.

Anyone know what the "oce" stands for?

Debug (ISAKMP & IPSec) Highlights

Jul 21 13:55:13.188 EDT: ISAKMP:(2597):SA authentication status: authenticated
Jul 21 13:55:13.236 EDT: ISAKMP:(2597):Old State = IKE_R_MM5 New State = IKE_P1_COMPLETE
Jul 21 13:55:13.356 EDT: IPSEC(create_sa): sa created,
Jul 21 13:55:13.356 EDT: IPSEC(create_sa): sa created,
Jul 21 13:55:13.356 EDT: %CRYPTO-5-SESSION_STATUS: Crypto tunnel is UP . Peer <spoke-physical-ip>:500 Id: spoke.domain.null
Jul 21 13:55:13.356 EDT: %DMVPN-7-CRYPTO_SS: Tunnel0-<hub-physical-ip> socket is UP
Jul 21 13:55:13.700 EDT: ISAKMP:(2597):Old State = IKE_QM_R_QM2 New State = IKE_QM_PHASE2_COMPLETE
Jul 21 13:56:49.601 EDT: IPSEC(cleanup_tun_decap_oce): unlock and null out Tunnel0 tun_decap_oce 86742E48 from ident 86FB990C
Jul 21 13:56:49.601 EDT: IPSEC(delete_sa): deleting SA,
Jul 21 13:56:49.601 EDT: IPSEC(delete_sa): deleting SA,
Jul 21 13:56:49.601 EDT: %CRYPTO-5-SESSION_STATUS: Crypto tunnel is DOWN. Peer <spoke-physical-ip>:500 Id: spoke.domain.null
Jul 21 13:56:49.601 EDT: ISAKMP:(2597):Input = IKE_MESG_FROM_IPSEC, IKE_PHASE2_DEL
Jul 21 13:56:49.605 EDT: ISAKMP:(2597):Input = IKE_MESG_INTERNAL, IKE_PHASE1_DEL

Note: A more complete debug output is attached.

General Observations (sh crypto isakmp sa, sh crypto ipsec sa)

ISAKMP SA reaches a state of QM_IDLE, and a status of ACTIVE. However, the SA is deleted and a new one is spawned within ~ minute.

IPSec SAs are negotiated on the hub and spoke. However, only the spoke has packet encaps, and only the hub has decaps. Wireshark confirms that the hub is not placing any ESP packets on the wire. The IPSec SAs are deleted, and new ones are spawned every ~1-1/2 minutes.

Show Command Output

hub#sh cry ipsec profile
IPSEC profile DMVPN
        Security association lifetime: 4608000 kilobytes/3600 seconds
        Responder-Only (Y/N): N
        PFS (Y/N): Y
        DH group: group2
        Transform sets={eni-xfm-des: { esp-des esp-sha-hmac } , eni-xfm-3des: { esp-3des esp-sha-hmac } ,}

hub#sh cry map
Crypto Map "Tunnel0-head-0" 65536 ipsec-isakmp
        Profile name: DMVPN
        Security association lifetime: 4608000 kilobytes/3600 seconds
        Responder-Only (Y/N): N
        PFS (Y/N): Y
        DH group: group2
        Transform sets={eni-xfm-des: { esp-des esp-sha-hmac } , eni-xfm-3des: { esp-3des esp-sha-hmac } ,}

Crypto Map "Tunnel0-head-0" 65537 ipsec-isakmp
        Map is a PROFILE INSTANCE.
        Peer = <spoke-physical-ip>
        Extended IP access list
            access-list permit gre host <hub-physical-ip> host <spoke-physical-ip>
        Current peer: <spoke-physical-ip>
        Security association lifetime: 4608000 kilobytes/3600 seconds
        Responder-Only (Y/N): N
        PFS (Y/N): Y
        DH group: group2
        Transform sets={eni-xfm-des: { esp-des esp-sha-hmac } , eni-xfm-3des: { esp-3des esp-sha-hmac } ,}
        Interfaces using crypto map Tunnel0-head-0:Tunnel0

hq-edg01#sh cry session detail
Crypto session current status

Interface: Tunnel0
Uptime: 00:00:10
Session status: UP-ACTIVE
Peer: <spoke-physical-ip> port 500 fvrf: (none) ivrf: (none)
      Phase1_id: spoke.domain.null
      Desc: (none)
IKE SA: local <hub-physical-ip>/500 remote <spoke-physical-ip>/500 Active
          Capabilities:(none) connid:2682 lifetime:23:59:47
IKE SA: local <hub-physical-ip>/500 remote <spoke-physical-ip>/500 Inactive
          Capabilities:(none) connid:2681 lifetime:0
IPSEC FLOW: permit 47 host <hub-physical-ip> host <spoke-physical-ip>
        Active SAs: 2, origin: crypto map
        Inbound: #pkts dec'ed 6 drop 0 life (KB/Sec) 4517257/3589
        Outbound: #pkts enc'ed 0 drop 0 life (KB/Sec) 4517258/3589

Hardware & IOS

c1811 (hub) - c181x-advipservicesk9-mz.124-24.T
c1711 (spoke) - c1700-advipservicesk9-mz.124-15.T9

Relevant crypto portions of the DMVPN configurations (hub/spoke) follow:

crypto isakmp policy 3
encr 3des
group 2
lifetime 86399

crypto isakmp identity hostname

crypto ipsec transform-set eni-xfm-3des esp-3des esp-sha-hmac
mode transport
crypto ipsec transform-set eni-xfm-des esp-des esp-sha-hmac
mode transport

crypto ipsec profile DMVPN
set security-association lifetime seconds 3600
set transform-set eni-xfm-des eni-xfm-3des
set pfs group2

interface Tunnel0
ip address <removed> 255.255.255.0
tunnel protection ipsec profile DMVPN

Note: NHRP, mGRE, and most other parameters have been snipped.

Any assistance would be appreciated.

Best Regards,
Mike

Michael Sullenberger · ‎08-10-2010

You are correct in your observation.

The previous p-pGRE interface (in your case) can get its information in to

the tunnel endpoint database (controls tunneling packets) even if the

p-pGRE tunnel is shutdown. Also in the code a GRE packet that is

destined to the router will check for a mathc with a p-pGRE tunnel before

mGRE tunnels. So the inbound GRE tunnel packets were getting "grabbed"

by the p-pGRE tunnel and then dropped.

If I really want a GRE tunnel to be "down" I will remove the 'tunnel source ...'.

If I want to have both tunnels up at the same time I do what you did, give

each one a different tunnel key or a different tunnel source.

Hope this helps explain what was happening.

Mike.

PS. You should be able to mark this as answered now.

View solution in original post

michael.leblanc · ‎07-21-2010

To All:

We've resolved the issue, and learned a lesson.

We were migrating from a P2P GRE to mGRE configuration. In doing so, we defined new tunnel interfaces for mGRE, and shutdown the P2P tunnel interfaces. We migrated configuration parameters we wished to retain, including the "tunnel key". We assumed erroneously, that we could do so given that the P2P tunnels were administratively shutdown. NOT!

On a hunch, we shutdown the mGRE tunnel interfaces, modified the tunnel key to a value different than that configured on the shutdown P2P tunnel interfaces, and then brought the mGRE interfaces out of shutdown. The tunnel came up, stayed up, NHRP registration took place, routes were exchanged, and we breathed a sigh of relief.

Best Regards,

Mike

michael.leblanc · ‎07-21-2010

To Anyone:

How do we mark our own post as "answered"?

Best Regards,

Mike

Michael Sullenberger · ‎08-10-2010

You are correct in your observation.

The previous p-pGRE interface (in your case) can get its information in to

the tunnel endpoint database (controls tunneling packets) even if the

p-pGRE tunnel is shutdown. Also in the code a GRE packet that is

destined to the router will check for a mathc with a p-pGRE tunnel before

mGRE tunnels. So the inbound GRE tunnel packets were getting "grabbed"

by the p-pGRE tunnel and then dropped.

If I really want a GRE tunnel to be "down" I will remove the 'tunnel source ...'.

If I want to have both tunnels up at the same time I do what you did, give

each one a different tunnel key or a different tunnel source.

Hope this helps explain what was happening.

Mike.

PS. You should be able to mark this as answered now.

michael.leblanc · ‎08-11-2010

Michael:

Thank you for the additional background info.

We'll make a point of removing the tunnel source when retaining a shutdown tunnel interface in future configs.

Best Regards,

Mike.