IPSec Troubleshooting: Problem Scenarios Part 2

Blog

Jul 20, 2011 10:29 PM
Jul 20th, 2011

This blog is in continuation with my If incase you are a new user then I would highly encourage you to go through first 2 blogs as well, to have a better understanding.

I also want to thank you for your incredible responses on other blogs.

In this part 2 will be discussing the following problem scenarios----

  • Routing  Issues (Reverse Route Injection)
  • DPD
  • Anti-Replay

Problem Scenario 1:


Routing Issues

User complains there is no traffic received through the IPSec tunnel. On further checking you find that IKE and IPSec SAs exist, but no end-end traffic; spoke shows its encrypting traffic however no decrpyt.

Check for IPSec SA on Hub Site (look for inbound and outbound SPIs, encr/decr counts)

HUB# sh crypto session remote 40.10.1.1 detail
Crypto session current status

Code: C - IKE Configuration mode, D - Dead Peer Detection    
K - Keepalives, N - NAT-traversal, X - IKE Extended Authentication

Interface: GigabitEthernet0/1
Profile: SPOKE10-PROF
Uptime: 00:01:49
Session status: UP-ACTIVE    
Peer: 40.10.1.1 port 500 fvrf: (none) ivrf: (none)
       Phase1_id: 40.10.1.1
       Desc: (none)
   IKE SA: local 30.3.1.1/500 remote 40.10.1.1/500 Active
           Capabilities:D connid:1029 lifetime:01:58:10
   IPSEC FLOW: permit ip 3.1.1.0/255.255.255.0 4.1.1.0/255.255.255.0
         Active SAs: 2, origin: crypto map
         Inbound:  #pkts dec'ed 9949 drop 60 life (KB/Sec) 4483560/1690
         Outbound: #pkts enc'ed 0 drop 0 life (KB/Sec) 4485046/1690

HUB# sh crypto ipsec sa peer 40.10.1.1

interface: GigabitEthernet0/1
     Crypto map tag: CMAP, local addr 30.3.1.1

    protected vrf: (none)
    local  ident (addr/mask/prot/port): (3.1.1.0/255.255.255.0/0/0)
    remote ident (addr/mask/prot/port): (4.1.1.0/255.255.255.0/0/0)
    current_peer 40.10.1.1 port 500
      PERMIT, flags={origin_is_acl,}
     #pkts encaps: 0, #pkts encrypt: 0, #pkts digest: 0
     #pkts decaps: 9949, #pkts decrypt: 9949, #pkts verify: 9949
     #pkts compressed: 0, #pkts decompressed: 0
     #pkts not compressed: 0, #pkts compr. failed: 0
     #pkts not decompressed: 0, #pkts decompress failed: 0
     #send errors 0, #recv errors 60

      local crypto endpt.: 30.3.1.1, remote crypto endpt.: 40.10.1.1
      path mtu 1500, ip mtu 1500, ip mtu idb GigabitEthernet0/1
      current outbound spi: 0xF6278D63(4129787235)

      inbound esp sas:
       spi: 0x16C58DD4(382045652)
         transform: esp-3des esp-sha-hmac ,
         in use settings ={Tunnel, }
         conn id: 27, flow_id: SW:27, crypto map: CMAP
         sa timing: remaining key lifetime (k/sec): (4483560/1659)
         IV size: 8 bytes
         replay detection support: Y
         Status: ACTIVE

      inbound ah sas:

      inbound pcp sas:

      outbound esp sas:
       spi: 0xF6278D63(4129787235)
         transform: esp-3des esp-sha-hmac ,
         in use settings ={Tunnel, }
         conn id: 28, flow_id: SW:28, crypto map: CMAP
         sa timing: remaining key lifetime (k/sec): (4485046/1657)
         IV size: 8 bytes
         replay detection support: Y
         Status: ACTIVE

      outbound ah sas:

      outbound pcp sas:

Check the Routes (for the Spoke protected networks)

HUB# sh ip route 4.1.1.1
% Network not in table
HUB#

HUB# sh ip cef 4.1.1.1
0.0.0.0/0, version 80, epoch 0, attached, default route handler
0 packets, 0 bytes
   via 0.0.0.0, 0 dependencies
     valid no route adjacency
HUB#

Check the Crypto Map for Reverse-Route Injection. This is needed for the Hub to inject a route for the Spoke protected subnets into its local routing table. The route is created when the IPSec SA is established. Since 12.4T, for this route to be created (based on the Crypto ACL) before the IPSec SA is established (so that the router can initiate the tunnel), we need the “reverse-route static” configuration.

In the VRF-Aware IPSec scenario, it is better to use the “reverse-route remote-peer <next-hop-gateway>” configuration under the crypto map.

Old Crytpo Map was----

crypto map CMAP 10 ipsec-isakmp
  set peer 40.10.1.1
  set transform-set TS
  set isakmp-profile SPOKE10-PROF
  match address SPOKE10-ACL

Lets add reverse route---

crypto map CMAP 10 ipsec-isakmp
  set peer 40.10.1.1
  set transform-set TS
  set isakmp-profile SPOKE10-PROF
  match address SPOKE10-ACL
reverse-route <static>

IPSEC(crypto_ipsec_sa_find_ident_head): reconnecting with the same proxies and peer 40.10.1.1
IPSEC(rte_mgr): VPN Route Event create SA based on crypto ACL in real time for 40.10.1.1
IPSEC(rte_mgr): VPN Route Refcount 1 GigabitEthernet0/1
IPSEC(rte_mgr): VPN Route Added 4.1.1.0 255.255.255.0 via 0.0.0.0 in IP DEFAULT TABLE with tag 0 distance 1

HUB# sh ip route 4.1.1.1
Routing entry for 4.1.1.0/24
   Known via "static", distance 1, metric 0
  Redistributing via ospf 1
   Advertised by ospf 1 subnets
   Routing Descriptor Blocks:
   * 40.10.1.1
       Route metric is 0, traffic share count is 1

HUB# sh ip cef 4.1.1.1 
4.1.1.0/24, version 83, epoch 0, cached adjacency 30.3.1.2
0 packets, 0 bytes
   via 40.10.1.1, 0 dependencies, recursive
     next hop 30.3.1.2, GigabitEthernet0/1 via 40.0.0.0/8

    valid cached adjacency

Complete Crypto Map

HUB# show crypto map

Crypto Map "CMAP" 10 ipsec-isakmp

        Peer = 40.10.1.1

        ISAKMP Profile: SPOKE10-PROF

        Extended IP access list SPOKE10-ACL

            access-list SPOKE10-ACL permit ip 3.1.1.0 0.0.0.255 4.1.1.0 0.0.0.255

        Current peer: 40.10.1.1

        Security association lifetime: 4608000 kilobytes/1800 seconds

        PFS (Y/N): N

        Transform sets={

                TS,

        }

        Reverse Route Injection Enabled

        Interfaces using crypto map CMAP:

                GigabitEthernet0/1

Problem Scenario 2:

DPD

This is a scenario where HUB keeps sending encrypted traffic, but it is not receiving any encrypted traffic from Spoke.  IKE and IPSec SAs are up.

Please perform the following steps------

Check if the Spoke is reachable (ping tunnel endpoint address)


HUB# ping 40.10.10.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 40.10.10.1, timeout is 2 seconds:
…..
Success rate is 0 percent (0/5)

No replies.

Check if Dead peer Detection is turned on

HUB# sh cry isa sa detail


Codes: C - IKE configuration mode, D - Dead Peer Detection
        K - Keepalives, N - NAT-traversal
        X - IKE Extended Authentication
        psk - Preshared key, rsig - RSA signature
        renc - RSA encryption
IPv4 Crypto ISAKMP SA

C-id    Local           Remote          I-VRF    Status      Encr  Hash  Auth  DH     Lifetime       Cap.

1035  30.3.1.1     40.10.1.1                     ACTIVE    3des  sha    psk      2      01:59:45    
        Engine-id:Conn-id =  SW:35

So Under Cap, its not listing anything, hence it is disabled.

Check the IPSec SA

HUB#sh cry ips sa

interface: GigabitEthernet0/1
     Crypto map tag: CMAP, local addr 30.3.1.1

    protected vrf: (none)
    local  ident (addr/mask/prot/port): (3.1.1.0/255.255.255.0/0/0)
    remote ident (addr/mask/prot/port): (4.1.1.0/255.255.255.0/0/0)
    current_peer 40.10.1.1 port 500
      PERMIT, flags={origin_is_acl,}
    #pkts encaps: 100, #pkts encrypt: 100, #pkts digest: 100
     #pkts decaps: 4, #pkts decrypt: 4, #pkts verify: 4
     #pkts compressed: 0, #pkts decompressed: 0
     #pkts not compressed: 0, #pkts compr. failed: 0
     #pkts not decompressed: 0, #pkts decompress failed: 0
     #send errors 0, #recv errors 0

      local crypto endpt.: 30.3.1.1, remote crypto endpt.: 40.10.1.1
      path mtu 1500, ip mtu 1500, ip mtu idb GigabitEthernet0/1
      current outbound spi: 0x8BDBBA86(2346433158)

      inbound esp sas:
       spi: 0x67C89AAD(1741200045)

We notice very few decaps, lets go ahead and configure DPD.

Configure DPD


            crypto isakmp keepalive 60 10
             crypto isakmp keepalive 60 periodic

If DPD had been configured earlier, then you would have seen following-----

HUB# sh cry isakmp peer de       
Peer: 40.10.1.1 Port: 500 Local: 30.3.1.1
  Phase1 id: 40.10.1.1
   flags:
  NAS Port: 0 (Normal) DPD information, struct 0x6727E0E8:
   Last_received: 237, dpd threshold (elapsed) 0
   my_last_seq_num: 0x5B72ECCC, peers_last_seq_num: 0x0
   sent_and_waiting: TRUE

  IKE SAs: 1 IPSec SA bundles: 1
   last_locker: 0x62FE32FC, last_last_locker: 0x0
   last_unlocker: 0x0, last_last_unlocker: 0x0

HUB# sh cry isa sa detail

  
Codes: C - IKE configuration mode, D - Dead Peer Detection
        K - Keepalives, N - NAT-traversal
        X - IKE Extended Authentication
        psk - Preshared key, rsig - RSA signature
        renc - RSA encryption
IPv4 Crypto ISAKMP SA

C-id  Local           Remote          I-VRF    Status Encr Hash Auth DH Lifetime  Cap.

1037  30.3.1.1        40.10.1.1                ACTIVE 3des sha  psk  2  01:59:43     D  
        Engine-id:Conn-id =  SW:37

PS: It shows D under Cap now.

ISAKMP:(1036):DPD incrementing error counter (4/5)
ISAKMP: set new node 1992211651 to QM_IDLE     
ISAKMP:(1036):Sending NOTIFY DPD/R_U_THERE protocol 1
         spi 1718567840, message ID = 1992211651
ISAKMP:(1036): seq. no 0x5B72ECCD
ISAKMP:(1036): sending packet to 40.10.1.1 my_port 500 peer_port 500 (R) QM_IDLE     
ISAKMP:(1036):Sending an IKE IPv4 Packet.
ISAKMP:(1036):purging node 1992211651
ISAKMP:(1036):Input = IKE_MESG_FROM_TIMER, IKE_TIMER_PEERS_ALIVE
ISAKMP:(1036):Old State = IKE_P1_COMPLETE  New State = IKE_P1_COMPLETE

ISAKMP:(1036):DPD incrementing error counter (5/5)
ISAKMP:(1036):peer 40.10.1.1 not responding!
ISAKMP:(1036):peer does not do paranoid keepalives.

ISAKMP:(1036):deleting SA reason "P1 errcounter exceeded (PEERS_ALIVE_TIMER)" state (R) QM_IDLE       (peer 40.10.1.1)

It is always better to use DPD instead of Periodic Keepalives. DPD works well in conjunction with IPSec HA – geographically distributed peers (multiple ‘set peer’ under crypto map), or HSRP adjacent peers (peer to VIP address).

Problem Scenario 3:


Anti-Replay Issues

Users complain that application is losing intermittent traffic, or that Voice quality through tunnel is bad. Check if the IPSec SA is showing anti-replay drops

HUB# sh cry ips sa peer 40.10.1.1 detail

interface: GigabitEthernet0/1
     Crypto map tag: CMAP, local addr 30.3.1.1

   protected vrf: (none)
    local  ident (addr/mask/prot/port): (3.1.1.0/255.255.255.0/0/0)
    remote ident (addr/mask/prot/port): (4.1.1.0/255.255.255.0/0/0)
    current_peer 40.10.1.1 port 500
      PERMIT, flags={origin_is_acl,}
     #pkts encaps: 2900, #pkts encrypt: 2900, #pkts digest: 2900
     #pkts decaps: 1909, #pkts decrypt: 1909, #pkts verify: 1909
     #pkts compressed: 0, #pkts decompressed: 0
     #pkts not compressed: 0, #pkts compr. failed: 0
    #pkts not decompressed: 0, #pkts decompress failed: 0
     #pkts no sa (send) 0, #pkts invalid sa (rcv) 0
     #pkts encaps failed (send) 0, #pkts decaps failed (rcv) 0
     #pkts invalid prot (recv) 0, #pkts verify failed: 0
     #pkts invalid identity (recv) 0, #pkts invalid len (rcv) 0
     #pkts replay rollover (send): 0, #pkts replay rollover (rcv) 0
     ##pkts replay failed (rcv): 1000
     #pkts internal err (send): 0, #pkts internal err (recv) 0

      local crypto endpt.: 30.3.1.1, remote crypto endpt.: 40.10.1.1
      path mtu 1500, ip mtu 1500, ip mtu idb GigabitEthernet0/1
      current outbound spi: 0xC37422AA(3279168170)

      inbound esp sas:
       spi: 0x135E76B1(324957873)
         transform: esp-3des esp-sha-hmac ,
         in use settings ={Tunnel, }
         conn id: 41, flow_id: SW:41, crypto map: CMAP
         sa timing: remaining key lifetime (k/sec): (4419198/860)
         IV size: 8 bytes
         replay detection support: Y
         Status: ACTIVE

So we can see there is a good number of packets replay failed in above show commands.

By Default IPSec Anti-Replay window size is 64. Hence, Packets received outside the window will be dropped. Normally Re-ordering of packets could happen due to QoS on the encrypting router (Spoke) or in the Transit Network.

In current Cisco IOS versions, the Anti-Replay window can be increased up to 1024, or diabled altogether


            
crypto ipsec security-association window-size < Size>

             crypto ipsec security-association replay disable

It is not recommended to disable anti-replay. Hence first try to fix the QoS issue in the network or encrypting router; give better QoS to Voice traffic, or use crypto LLQ; then try to increase the anti-replay window size by above mentioned command.

That sums up my problem scenarios blog. Hope you liked these scenarios and you will be able to implement these best practices in your infrastructure to have a seamless network.

Looking forward to your comments and inputs and also please let me know on what topic you want to see future blogs.

Appreciate your time and contribution on CSC

Average Rating: 5 (3 ratings)

Comments

Actions

Login or Register to take actions

This Blog

Posted July 20, 2011 at 10:29 PM
Stats:
Comments:2 Avg. Rating:5
Views:8314   
Shares:0

Related Content

Blogs Leaderboard