IPSec important Debugging and logging

Blog

May 1, 2011 7:44 AM
May 1st, 2011
/* Style Definitions */ table.MsoNormalTable {mso-style-name:"Table Normal"; mso-tstyle-rowband-size:0; mso-tstyle-colband-size:0; mso-style-noshow:yes; mso-style-priority:99; mso-style-qformat:yes; mso-style-parent:""; mso-padding-alt:0in 5.4pt 0in 5.4pt; mso-para-margin-top:0in; mso-para-margin-right:0in; mso-para-margin-bottom:10.0pt; mso-para-margin-left:0in; line-height:115%; mso-pagination:widow-orphan; font-size:11.0pt; font-family:"Calibri","sans-serif"; mso-ascii-font-family:Calibri; mso-ascii-theme-font:minor-latin; mso-hansi-font-family:Calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"Times New Roman"; mso-bidi-theme-font:minor-bidi;}

I thought of sharing ipsec debugging and troubleshooting steps with everyone. Being in VPN technology we explain this to many of our customers and thought of discussing it here on our support forum as well.

What is IPSEC?

IPSec stands for IP Security and the standard definition of IPSEC is--

“A security protocol in the network layer will be developed to provide cryptographic security services that will flexibly support combinations of authentication, integrity, access control, and confidentiality” (IETF)

It is a standard for privacy, integrity and authenticity.

IPSEC Protocol Architecture

IPSEC is a combination of three primary protocols ESP(protocol 50), AH(protocol 51) and IKE(UDP 500)

Authentication: Authentication Header (AH) and  Encapsulating Security Payload (ESP)

Integrity: Encapsulating Security Payload (ESP)

Confidentiality: Encapsulating Security Payload  (ESP)

Bringing it all together: Internet key Exchange (IKE)

IPSEC is implemented in the following five stages:

Decision to use IPSEC between two end points across internet

Configuration of the two gateways between the end points to support IPSEC

Initiation of an IPSEC tunnel between the two gateways due to ‘interesting traffic’

Negotiation of IPSEC/IKE parameters between the two gateways

Passage of encrypted traffic

IPSec Troubleshooting Steps

  • Check  for interesting traffic to initiate tunnel, check crypto ACLs for hit      counts

        If not, verify Routing (static or RRI)

  • Verify  if IKE SA is up (QM_Idle) for that peer

        If not, verify for matching Pre-shared keys

        Verify that the IKE policies (encr, auth, DH) are matching

        Verify for matching IKE Identities

  • Verify  if IPSec SAs are up (Inbound and Outbound SPIs)

        If not, verify for matching IPSec transform sets

        Verify for mirrored crypto ACLs on each side

        Verify that the Crypto Map is applied on the right interface

  • Turn on IKE/IPSec debugs

IPSec Show Commands

  • To show IKE SA information:

        show crypto isakmp sa <vrf> [detail]

        show crypto isakmp peer <ip-addr>

  • To show IPSec SA information:

        show crypto ipsec sa [ address | detail | interface | map | per | vrf ]

  • To  show IKE and IPSec information together :

        show crypto session  [ fvrf | group | ivrf ] username | detail ]

        show crypto engine connection active

Cisco IOS IPSec Debugging

  • These are the current IKE/IPSec debugs available; the highlighted ones are the      most useful typically
  • Make  sure to use Crypto Conditional Debugs when trying to troubleshoot      production routers

        debug crypto isakmp

        debug crypto isakmp error

        debug crypto isakmp ha          

        debug crypto ipsec      

        debug crypto ipsec error         

        debug crypto routing       

        debug crypto ha        

        debug crypto engine error      

        debug crypto engine packet     

Crypto Conditional Debugging

We can use crypto conditional debugging when we are troubleshooting live networks and specially where there are multiple tunnels running on the device.

  • The crypto conditional debug CLIs—debug crypto condition, debug crypto condition unmatched, and show crypto debug-condition— allow you to specify conditions (filter values) in which to generate and display debug messages related only to the specified conditions
  • The router will perform conditional debugging only after at least one of the global crypto debug commands—debug crypto isakmp, debug crypto ipsec, or debug crypto engine—has been enabled; thi s requirement helps to ensure that the performance of the router will not be impacted when conditional debugging is not being used
  • To  enable crypto conditional debugging:

        debug crypto condition <cond-type> <cond-value>

        debug crypto { isakmp | ipsec | engine }

  • To view crypto condition debugs that have been enabled:

        show crypto debug-condition [ all | peer | fvrf | ivrf | isakmp | username | connid | spi ]

  • To disable crypto condition debugs:

        debug crypto condition reset

Crypto Conditional Debugging

Fvrf

The name   string of a virtual private network (VPN) routing and forwarding (VRF)   instance; relevant debug messages will be shown if the current IPSec   operation uses this VRF instance as its front-door VRF (FVRF)

ivrf

The name   string of a VRF instance; relevant debug messages will be shown if the   current IPSec operation uses this VRF instance as its inside VRF (IVRF)

isakmp   profile

The name   string of the isakmp profile to be matched against for debugging

Local   ipv4

The ip   address string of the local IKE endpoint

Peer   group

A ezvpn   group name string; relevant debug messages will be shown if the peer is using   this group name as its identity

Peer   ipv4

A single   IP address; relevant debug messages will be shown if the current IPSec   operation is related to the IP address of this peer

Peer   subnet

A subnet   and a subnet mask that specify a range of peer IP addresses; relevant debug   messages will be shown if the IP address of the current IPSec peer falls into   the specified subnet range

Peer   hostname

A fully   qualified domain name (FQDN) string; relevant debug messages will be shown if   the peer is using this string as its identity

username

The   username string (XAuth username or PKI-aaa username obtained from a   certificate)

Clearing VPN Tunnel

  

  • To clear IKE Phase ( Phase 1)

         clear crypto isakmp sa

  • To clear IPSEC Phase (Phase2)
                  –          clear crypto ipsec sa

Crypto Logging

Two crypto logging enhancements were introduced in recent Cisco IOS images

Hub(config)# crypto logging ?

          ezvpn                  ezvpn logging enable/disable

          session               logging up/down session

        Crypto logging session, introduced in 12.3(14)T, displays tunnel up/down messages:

        %CRYPTO-5-SESSION_STATUS: Crypto tunnel is UP  .  Peer 40.10.1.1:500       Id: 40.10.1.1

        %CRYPTO-5-SESSION_STATUS: Crypto tunnel is DOWN.  Peer 40.10.1.1:500       Id: 40.10.1.1

        Crypto logging ezvpn, introduced in 12.4(4)T, displays EasyVPN connection messages

        %CRYPTO-5-SESSION_STATUS: Crypto tunnel is DOWN.  Peer 2.2.2.2:500 f_vrf:  FVRF1     Id: cisco

        %CRYPTO-6-EZVPN_CONNECTION_DOWN: (Server)  Mode=NEM  Client_type=CISCO_IOS  User=  Group=cisco  Client_public_addr=2.2.2.2  Server_public_addr=1.1.1.2  f_vrf=FVRF1

        %CRYPTO-5-SESSION_STATUS: Crypto tunnel is UP  .  Peer 2.2.2.2:500 f_vrf:  FVRF1     Id: cisco

        %CRYPTO-6-EZVPN_CONNECTION_UP: (Server)  Mode=NEM  Client_type=CISCO_IOS  User=  Group=cisco  Client_public_addr=2.2.2.2  Server_public_addr=1.1.1.2  f_vrf=FVRF1 

That’s all from my side today.

I am thinking of coming up with few known issues or scenarios in my next blog, hence looking forward to your inputs and feedbacks. Thanks

Average Rating: 5 (2 ratings)

Comments

network770 Tue, 05/31/2011 - 18:43 (reply to athukral)

HI Ankur,

We have a lot of L2L and remote access ipsec sessiosn and when tunnels go down or not coming up when we build them we often run into problems and don't know how to determine what causing it.  can you post a list of messages one should expect to see in a given debug and how to solve it.  We open a tac case once and the engineer ran a debug on crypto isakmp and ipsec and looking at the debug logs he was able to pin point what the problem is, we are looking for such a document

josemed Wed, 06/01/2011 - 12:12 (reply to network770)

Hello Ronni,

Here is some info I was able to find that you might wanna keep handy.

http://www.cisco.com/en/US/docs/ios/12_2/debug/command/reference/dbfclns.html#wp1018145

http://www.cisco.com/en/US/tech/tk583/tk372/technologies_tech_note09186a00800949c5.shtml

http://www.cisco.com/en/US/products/ps6120/products_tech_note09186a00807e0aca.shtml#Solution2

This is also some good information:

/* Style Definitions */ table.MsoNormalTable {mso-style-name:"Table Normal"; mso-tstyle-rowband-size:0; mso-tstyle-colband-size:0; mso-style-noshow:yes; mso-style-priority:99; mso-style-qformat:yes; mso-style-parent:""; mso-padding-alt:0in 5.4pt 0in 5.4pt; mso-para-margin:0in; mso-para-margin-bottom:.0001pt; mso-pagination:widow-orphan; font-size:11.0pt; font-family:"Calibri","sans-serif"; mso-ascii-font-family:Calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:Calibri; mso-fareast-theme-font:minor-latin; mso-hansi-font-family:Calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"Times New Roman"; mso-bidi-theme-font:minor-bidi; mso-fareast-language:EN-US;}

ISAKMP Negotiations States

These are the possible ISAKMP negotiation states on an ASA firewall. ISAKMP stands for: The Internet Security Association and Key Management Protocol

ASA ISAKMP STATES

    * MM_WAIT_MSG2

      Initial DH public key sent to responder. Awaiting initial contact reply from other side.

      If stuck here it usually means the other end is not responding. This could be due to no route to the far end or the far end does not have ISAKMP enabled on the outside or the far end is down.

    * MM_WAIT_MSG3

      Both peers have agreed on the ISAKMP policies. Awaiting exchange of keyring information.

      Hang up’s here may be due to mismatch device vendors, a router with a firewall in the way, or even ASA version mismatches.

    * MM_WAIT_MSG4

      In this step the pre-share key hashes are exchanged. They are not compared or checked, only sent. If one side sends a key and does not receive a key back, this is where the tunnel will fail.

      I have seen the tunnel fail at this step due to the remote side having the wrong Peer IP address. Hang up’s here may also be due to mismatch device vendors, a router with a firewall in the way, or even ASA version mismatches.

    * MM_WAIT_MSG5

      This step is where the devices exchange pre-shared keys.

      If the pre-shared keys do not match it will stay at this MSG. I have also seen the tunnel stop here when NAT Traversal was on when it needed to be turned off.

    * MM_WAIT_MSG6

      This step is where the devices exchange pre-shared keys.

      If the pre-shared keys do not match it will stay at this MSG. I have also seen the tunnel stop here when NAT Traversal was on when it needed to be turned off.

      However, if the state goes to MSG6 then the ISAKMP gets reset that means phase 1 finished but phase 2 failed. Check that IPSEC settings match in phase 2 to get the tunnel to MM_ACTIVE.

    * AM_ACTIVE / MM_ACTIVE

      The ISAKMP negotiations are complete. Phase 1 has successfully completed.

PIX ISAKMP STATES

    * MM_NO_STATE

      ISAKMP SA has been created but nothing else has happened yet.

    * MM_SA_SETUP

      The peers have agreed on parameters for the ISAKMP SA.

    * MM_KEY_EXCH

      The peers have exchanged Diffie-Hellman public keys and have generated a shared secret. The I SAKMP SA remains unauthenticated.

    * MM_KEY_AUTH

      The ISAKMP SA has been authenticated. If the router initiated this exchange, this state trans itions immediately to QM_IDLE and a Quick mode exchange begins.

    * AG_NO_STATE

      The ISAKMP SA has been created but nothing else has happened yet.

    * AG_INIT_EXCH

      The peers have done the first exchange in Aggressive mode but the SA is not authenticated.

    * AG_AUTH

      The ISAKMP SA has been authenticated. If the router initiated this exchange, this state transitions immediately to QM_IDLE and a Quick mode exchange begins.

    * QM_IDLE

      The ISAKMP negotiations are complete. Phase 1 successfully completed. It remains authenticated with its peer and may be used for subsequent Quick mode exchanges.

Errors With ISAKMP Or Phase 1 Establishing

A very common problem is phase 1 not establishing correctly.

    * Verify ISAKMP parameters match exactly.

    * Verify pre-shared-keys match exactly.

    * Check that each side has a route to the peer address that you are trying to form a tunnel with.

    * Verify ISAKMP/crypto map is enabled on the outside interfaces.

    * Is UDP port 500 open on the outside ACL?

    * Some situations require that UDP port 4500 is open for the outside.

    * Capture on the outside interface of each of the VPN termination devices

    * If the packet is not getting to the VPN termination, check with the ISP and request them to sniff the packets that are getting to the ISP modem. In      this way, you can confirm if the packet is even getting to the ISP modem

---josemed

athukral Wed, 06/01/2011 - 22:58 (reply to josemed)

Thanks Gustavo, very informative addition

Ronni, I am in middle of writing my next blog, where i will be taking up couple of scenrios , hope that will be helpful!

Thanks

network770 Thu, 06/02/2011 - 11:31 (reply to athukral)

Gustavo, thanks a million for all this. For a while I've been looking for a document that describes issues with isakmp and ipsec, what error messages I should expect to see and how to resolve it, so that is all great.  Ankur, if you have more info to add to this I would appreciated it greatly, I can formalize this doc and post it on the supportforms for others. Troubleshooting ipsec is not easy and having a reference document like this makes it easier

pavan.katakam Tue, 06/07/2011 - 22:44

Awesome Ankur! You have explained in a very lucid way. Even a beginner can understand it Thanks for the post. Looking forward for the scenarios

pgoutham Thu, 06/16/2011 - 22:19

This is a wonderful attempt to bridge the world of IPSec Geekstuff into plain english hints.

This would take away the difficulty for several engineers who often fret the world of IPsec including me.

Awesome attempt Ankur, keep it up !

marioderosa2008 Fri, 11/01/2013 - 02:21

can i ask what minimum ASA software you need to have to do the conditional debugging?

I cannot see the condition option on my ASA and I am running Cisco Adaptive Security Appliance Software Version 7.2(3)..

Thanks

Mario

Actions

Login or Register to take actions

This Blog

Posted May 1, 2011 at 7:44 AM
Stats:
Comments:13 Avg. Rating:5
Views:28672   
Shares:3

Related Content

Blogs Leaderboard