L2L VPN- difference in number of encaps and decaps

saifuddin.miyaji · ‎10-30-2010

I have got L2L tunnel between 2 sites and its experiencing heavy packet loss. Internet connection looks great. The interesting part is that 'show crypto ipsec sa' shows that there is considerable difference in the number of packets encryp/decrypt and encaps/decaps. And the number of packet drops is very close to the difference of the encaps/decaps or encrypt/decrypt

The ping from one site to another on the interesting traffic looks like this:

VoiceRTR-Qr#ping 10.10.7.2 source 10.15.7.2 repeat 100

Type escape sequence to abort.
Sending 100, 100-byte ICMP Echos to 10.10.7.2, timeout is 2 seconds:
Packet sent with a source address of 10.15.7.2
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!
.!!.!!!.!!.!!!.!!!.!!.!!!.!!.!
Success rate is 90 percent (90/100), round-trip min/avg/max = 68/95/128 ms

and the output of the ' show crypto ipsec sa' looks like this:

ASA-S-1# sh cry ipsec sa peer xx.xx.xx.xx

peer address: xx.xx.xx.xx
Crypto map tag: SYSTEM_DEFAULT_CRYPTO_MAP, seq num: 65535, local addr: yy.yy.yy.yy

      local ident (addr/mask/prot/port): (10.10.0.0/255.255.0.0/0/0)
      remote ident (addr/mask/prot/port): (10.15.0.0/255.255.0.0/0/0)
      current_peer: xx.xx.xx.xx

      #pkts encaps: 79, #pkts encrypt: 79, #pkts digest: 79
      #pkts decaps: 65, #pkts decrypt: 65, #pkts verify: 65
      #pkts compressed: 0, #pkts decompressed: 0
      #pkts not compressed: 79, #pkts comp failed: 0, #pkts decomp failed: 0
      #pre-frag successes: 0, #pre-frag failures: 0, #fragments created: 0
      #PMTUs sent: 0, #PMTUs rcvd: 0, #decapsulated frgs needing reassembly: 0
      #send errors: 0, #recv errors: 0

      local crypto endpt.: yy.yy.yy.yy/4500, remote crypto endpt.: xx.xx.xx.xx/4500
      path mtu 1500, ipsec overhead 66, media mtu 1500
      current outbound spi: C6738AB6
      current inbound spi : 3452F7D5

    inbound esp sas:
      spi: 0x3452F7D5 (877852629)
         transform: esp-3des esp-sha-hmac no compression
         in use settings ={L2L, Tunnel, NAT-T-Encaps, PFS Group 1, }
         slot: 0, conn_id: 9687040, crypto-map: SYSTEM_DEFAULT_CRYPTO_MAP
         sa timing: remaining key lifetime (kB/sec): (4373954/25087)
         IV size: 8 bytes
         replay detection support: Y
         Anti replay bitmap:
          0xFFFFFFFF 0xFFFFFFFF
    outbound esp sas:
      spi: 0xC6738AB6 (3329460918)
         transform: esp-3des esp-sha-hmac no compression
         in use settings ={L2L, Tunnel, NAT-T-Encaps, PFS Group 1, }
         slot: 0, conn_id: 9687040, crypto-map: SYSTEM_DEFAULT_CRYPTO_MAP
         sa timing: remaining key lifetime (kB/sec): (4373936/25086)
         IV size: 8 bytes
         replay detection support: Y
         Anti replay bitmap:
          0x00000000 0x00000001

Now, can someone shed some light on the probable reasons for some packets not getting encaps and encrypt?

saif

Maykol Rojas · ‎10-30-2010

Hello Saif

I think the best way to troubleshoot this would be by discarting first an ISP issue. Are you having the same packet loss over the internet? If you try to ping 4.2.2.2 repeat 500 do you get the same results? Are the interfaces having any errors?

You can also check the "show asp drop" and also take a capture on the ASA device as asp drop all to check if you are able to see any drops related to the tunnel.

Hope this helps.

Mike

saifuddin.miyaji · ‎10-31-2010

Maykol, as i said in the initial post, the Internet connection looks great. A ping with 1000 packets does not have even a single drop. Also, ' sh asp drop' did not give me any useful info. I could not find any correlation between the asp counters and packet drops, neither does the packet capture helped here.

saif

Srikanth K. S · ‎10-30-2010

Hi Saif,

I suggest that we do the similar test by initiating the ping from the other side (I presume the ASA side, the site to site being between ASA and the VoiceRTR-Qr) and check if we see any packet drops and the encaps and decaps are consistent.

There could be an issue where in these intermediate packets are not even reaching the router.

Is there any layer 2 or layer 3 switch behind the Router? If there is a device behind the router the problem could be with that device as well.

To check with and isolate the problem, what that what we can do is we can create an acl saying,

Router(config)# ip access-list extended Test

Router(config-ext-nacl)# permit ip host 10.15.7.2 host 10.10.7.2

Router(config-ext-nacl)# permit ip any any (Please include this statement if the device is in production, as it should be applied on the interface and will deny all other traffic if not given).

Apply the above to the inside interface of the router where 10.15.7.2 is connected.

Router(config-if)# ip access-group Test in

Now initiate the ping again

ping 10.10.7.2 source 10.15.7.2 repeat 100

Do a sh access-list Test

Now, if you are seeing hit counts as 100 in the access-list and packet drops are happening and still encaps and decaps are not consistent, then there could be a problem with the configuration in the router and we need to check that.

In case we do not see the 100 hits in the above access-list, then the packets are not reaching the router and are getting dropped somewhere on the way to the router.

Please try and let me know how it goes.

Regards,

Srikanth K S.

Praveena Shanubhogue · ‎10-31-2010

Hi Saif,

On the router, try disabling cef ("no ip cef" in global conf mode) and then repeat the pings. Let me know if you see the same issues!

Regards,

Praveen