Disclaimer: This content has been initially created by Alex Honore, Graham Barlet, Raffaele Brancaleoni from Cisco.
This document is indetnded to show troubleshooting order of operation when troubleshooting DMVPN setup.
It will also discuss responsibilites of different components, but not in detail.
It is intended as a troubleshooting framework and a referance on which commands to be used.
Please remember, that you should always start troubleshooting of DMVPN on spokes and not hubs.
Impact of changes on spokes is limited to particular location, while changes/troubleshooting on hub side are potetially impacting entire DMVPN cloud.
Using a recent revision of IOS software helps overcome a lot of problems with monitoring and troubleshooting.
Add this command to running configuration on both sides of problematic tunnel:
crypto logging session
allows to see indepth information about failures and potential reasons.
This section describes step by step approach to troubleshooting DMVPN.
Using this framework allows to easily verify offending component and apply remediation.
NHRP is the trigger for spoke to hub initial tunnel establishment.
Key components at this stage can be verfied in the configuration:
show run interface tunnel X
Things to verify:
IKE provides keying material to IPsec and is therefor a preamble to it.
It is during IKE negotiatin that calls are made to PKI infrastructure.
Understand what state IKE is at the moment.
show crypto isakmp sa
MM_NO_STATE will mean a failure or expired SA.
MM_KEY_EXCH will signifiy that we are in process or stuck during authentication (PKI or pre-shared).
PKI is component responsible for the authentication of peers in IKE negotiation.
To accomplish its task, it requites the retrieval and processing of revocation information (CRL).
Check the time:
Check the validity of certificates on both sides:
show crypto pki cert
Check if you have a recent version of CRL downloaded:
show cry pki crls
Check availability of CDP:
telnet IP_OR_HOSTNAME 80
GET /<name_of_crl_file> HTTP/1.0
IPsec by itself is stateless, but encrypted traffic emission and reception capabilities must be confirmed (think about IP protocol type 50/51 filtering in transport network for ex.)
Verfies if both IKE and IPsec are up.
show crypto session [detail]
For working session you will see TWO active SAs per peer. One SA for inbound traffic and one SA for outbound traffic.
Verify if tunnel is passing traffic.
show crypto ipsec sa peer IP
Working scenario - encaps and decaps section counters are non-zero AND increasing.
Provide multiprotocol support capability & multicast support to IPsec along with simplifying configs. Only fails if tunnel keys between GRE endpoints are different
On both hub and spoke
show tunnel endpoint tunnel X
On spoke this information should be populated by static NHRP mapping.
NHRP performs the initial registration to Hub which forms the basis of DMVPN and is at the root of the spoke to spoke dynamic IPsec tunnels construction
On spoke, check if you see responses from NHS.
show ip nhrp nhs
On hub , verify if dynamic NHRP entiries were populated.
show ip nhrp tunnel X
show ip nhrp multicast
Propagate the necessary routing information between hubs and spokes
It's hard to point to one set of commands to troubleshoot this - DMVPN supports multiple routing protocols.
In any case you need to verify if a particular adjacancy/neighborship came up correctly and whether you are sending and receiving correct prefixes.
BGP is by far the best protocol to scale in DMVPN networks. With recent additions to it (like dynamic listener) we are able to create working configuration hassle-free.
show ip bgp [vpnv4 vrf VRF_NAME] summary
show ip bgp [vpnv4 vrf VRF_NAME]
show ip bgp [vpnv4 vrf VRF_NAME] neighbor IP_ADDRESS routes
show ip bgp [vpnv4 vrf VRF_NAME] neighbor IP_ADDRESS advertised
show ip router bgp
show ip eigrp neighbor
show ip eigrp topology
show ip router eigrp
Thanks for this great contribution Marcin!
Extremely easy to digest treatment of a relatively complex topic. Thanks!!!