Anyone have any suggestions ?

roadhouse1387 · ‎09-17-2014

Hi All,

I have an IKEv1 IPSEC L2L tunnel setup between a 5525x ASA and a 5510 ASA both running 9.1(5), and am having trouble with asymmetric decryption at one side of the tunnel.

I have a very simple one line crypto ACL which permits a /24 subnet to talk to a /20 and vice versa, I have checked all the relevant NO NAT's are in place, they are, as is the routing.

The ISAKMP SA's are all in place and look good (sanitized).

IKEv1 SAs:

Active SA: 1
Rekey SA: 0 (A tunnel will report 1 Active and 1 Rekey SA during rekey)
Total IKE SA: 1

1   IKE Peer: 1.1.1.1
    Type    : L2L             Role    : responder
    Rekey   : no              State   : MM_ACTIVE

There are no IKEv2 SAs

On the 5510, I can see that arriving traffic is being decrypted from the 5525x ok, and is being encrypted outbound toward the 5525x as well, again this looks good.

interface: outside
Crypto map tag: outside_map, seq num: 1, local addr: 2.2.2.2

      access-list outside_cryptomap extended permit ip 10.20.0.0 255.255.255.0 10.1.64.0 255.255.240.0
      local ident (addr/mask/prot/port): (10.20.0.0/255.255.255.0/0/0)
      remote ident (addr/mask/prot/port): (10.1.64.0/255.255.240.0/0/0)
      current_peer: 1.1.1.1

      #pkts encaps: 54, #pkts encrypt: 54, #pkts digest: 54
      #pkts decaps: 411, #pkts decrypt: 411, #pkts verify: 411
      #pkts compressed: 0, #pkts decompressed: 0
      #pkts not compressed: 54, #pkts comp failed: 0, #pkts decomp failed: 0
      #pre-frag successes: 0, #pre-frag failures: 0, #fragments created: 0
      #PMTUs sent: 0, #PMTUs rcvd: 0, #decapsulated frgs needing reassembly: 0
      #TFC rcvd: 0, #TFC sent: 0
      #Valid ICMP Errors rcvd: 0, #Invalid ICMP Errors rcvd: 0
      #send errors: 0, #recv errors: 0

      local crypto endpt.: 2.2.2.2/0, remote crypto endpt.: 1.1.1.1/0
      path mtu 1500, ipsec overhead 58(36), media mtu 1500
      PMTU time remaining (sec): 0, DF policy: copy-df
      ICMP error validation: disabled, TFC packets: disabled
      current outbound spi: 1CDEA655
      current inbound spi : A60AA787

    inbound esp sas:
      spi: 0xA60AA787 (2785716103)
         transform: esp-3des esp-sha-hmac no compression
         in use settings ={L2L, Tunnel, PFS Group 2, IKEv1, }
         slot: 0, conn_id: 278528, crypto-map: outside_map
         sa timing: remaining key lifetime (kB/sec): (4373917/27576)
         IV size: 8 bytes
         replay detection support: Y
         Anti replay bitmap:
          0xFFFFFFFF 0xFFFFFFFF
    outbound esp sas:
      spi: 0x1CDEA655 (484353621)
         transform: esp-3des esp-sha-hmac no compression
         in use settings ={L2L, Tunnel, PFS Group 2, IKEv1, }
         slot: 0, conn_id: 278528, crypto-map: outside_map
         sa timing: remaining key lifetime (kB/sec): (4373985/27576)
         IV size: 8 bytes
         replay detection support: Y
         Anti replay bitmap:
          0x00000000 0x00000001

The output of the show crypto ipsec accelerator stats on the 5510 gives this... which seems to show that stuff is going in and out ok via the hardware engine in slot 1

Crypto Accelerator Status

-------------------------
[Capability]
   Supports hardware crypto: True
   Supports modular hardware crypto: False
   Max accelerators: 2
   Max crypto throughput: 170 Mbps
   Max crypto connections: 250
[Global Statistics]
   Number of active accelerators: 2
   Number of non-operational accelerators: 0
   Input packets: 523405
   Input bytes: 111848159
   Output packets: 230595
   Output error packets: 0
   Output bytes: 230240311

[Accelerator 0]
   Status: OK
   Software crypto engine
   Slot: 0
   Active time: 986957 seconds
   Total crypto transforms: 13635
   Total dropped packets: 0
   [Input statistics]
      Input packets: 0
      Input bytes: 51856
      Input hashed packets: 0
      Input hashed bytes: 0
      Decrypted packets: 0
      Decrypted bytes: 51856
   [Output statistics]
      Output packets: 0
      Output bad packets: 0
      Output bytes: 109776
      Output hashed packets: 0
      Output hashed bytes: 0
      Encrypted packets: 0
      Encrypted bytes: 109984
   [Diffie-Hellman statistics]
      Keys generated: 0
      Secret keys derived: 0
   [RSA statistics]
      Keys generated: 1
      Signatures: 6
      Verifications: 1
      Encrypted packets: 1
      Encrypted bytes: 28
      Decrypted packets: 1
      Decrypted bytes: 256
   [ECDSA statistics]
      Keys generated: 12
      Signatures: 12
      Verifications: 15
   [SSL statistics]
      Outbound records: 0
      Inbound records: 0
   [RNG statistics]
      Random number requests: 71
      Random number request failures: 0
   [HMAC statistics]
      HMAC requests: 3222

[Accelerator 1]
   Status: OK
   Encryption hardware device : Cisco ASA-55xx on-board accelerator (revision 0x0)
                             Boot microcode        : CN1000-MC-BOOT-2.00
                             SSL/IKE microcode     : CNLite-MC-SSLm-PLUS-2_05
                             IPSec microcode       : CNlite-MC-IPSECm-MAIN-2.09
   Slot: 1
   Active time: 986963 seconds
   Total crypto transforms: 1253215
   Total dropped packets: 0
   [Input statistics]
      Input packets: 523436
      Input bytes: 128675042
      Input hashed packets: 357595
      Input hashed bytes: 81336891
      Decrypted packets: 523439
      Decrypted bytes: 96027422
   [Output statistics]
      Output packets: 230669
      Output bad packets: 0
      Output bytes: 442975336
      Output hashed packets: 32674
      Output hashed bytes: 9898376
      Encrypted packets: 230677
      Encrypted bytes: 228312109
   [Diffie-Hellman statistics]
      Keys generated: 216
      Secret keys derived: 55
   [RSA statistics]
      Keys generated: 0
      Signatures: 1
      Verifications: 1
      Encrypted packets: 2
      Encrypted bytes: 63
      Decrypted packets: 2
      Decrypted bytes: 512
   [ECDSA statistics]
      Keys generated: 0
      Signatures: 0
      Verifications: 0
   [SSL statistics]
      Outbound records: 198027
      Inbound records: 165862
   [RNG statistics]
      Random number requests: 235
      Random number request failures: 0
   [HMAC statistics]
      HMAC requests: 99983

on the 5525x side, I see traffic leaving outbound is encrypted ok, but nothing is being decrypted inbound, from the 5510. There is only one IPSEC VPN configured on each ASA and in essence , there isn't much to go wrong it would appear.

The crypto IPSEC SA on the 5525x looks like this... again ok, apart from the obvious lack of inbound decryption...

interface: outside
Crypto map tag: outside_map, seq num: 1, local addr: 1.1.1.1

      access-list outside_cryptomap_1 extended permit ip 10.1.64.0 255.255.240.0 10.20.0.0 255.255.255.0
      local ident (addr/mask/prot/port): (10.1.64.0/255.255.240.0/0/0)
      remote ident (addr/mask/prot/port): (10.20.0.0/255.255.255.0/0/0)
      current_peer: 2.2.2.2

      #pkts encaps: 607, #pkts encrypt: 607, #pkts digest: 607
      #pkts decaps: 0, #pkts decrypt: 0, #pkts verify: 0
      #pkts compressed: 0, #pkts decompressed: 0
      #pkts not compressed: 607, #pkts comp failed: 0, #pkts decomp failed: 0
      #pre-frag successes: 0, #pre-frag failures: 0, #fragments created: 0
      #PMTUs sent: 0, #PMTUs rcvd: 0, #decapsulated frgs needing reassembly: 0
      #TFC rcvd: 0, #TFC sent: 0
      #Valid ICMP Errors rcvd: 0, #Invalid ICMP Errors rcvd: 0
      #send errors: 0, #recv errors: 0

      local crypto endpt.: 1.1.1.1/0, remote crypto endpt.: 2.2.2.2/0
      path mtu 1500, ipsec overhead 58(36), media mtu 1500
      PMTU time remaining (sec): 0, DF policy: copy-df
      ICMP error validation: disabled, TFC packets: disabled
      current outbound spi: A60AA787
      current inbound spi : 1CDEA655
<--- More --->

    inbound esp sas:
      spi: 0x1CDEA655 (484353621)
         transform: esp-3des esp-sha-hmac no compression
         in use settings ={L2L, Tunnel, PFS Group 2, IKEv1, }
         slot: 0, conn_id: 495616, crypto-map: outside_map
         sa timing: remaining key lifetime (kB/sec): (3915000/26998)
         IV size: 8 bytes
         replay detection support: Y
         Anti replay bitmap:
          0x00000000 0x00000001
    outbound esp sas:
      spi: 0xA60AA787 (2785716103)
         transform: esp-3des esp-sha-hmac no compression
         in use settings ={L2L, Tunnel, PFS Group 2, IKEv1, }
         slot: 0, conn_id: 495616, crypto-map: outside_map
         sa timing: remaining key lifetime (kB/sec): (3914877/26998)
         IV size: 8 bytes
         replay detection support: Y
         Anti replay bitmap:
          0x00000000 0x00000001

And this is the show ipsec accelerator from the 5525x... It shows the same, apart from on accelerator 1, it actually appears to show inbound packets are being received, but none of them are subject to hashing, Im not sure if this is the problem itself or just symptomatic of something else, but it does suggest to me that the traffic is arriving ok, but just not being decrypted properly for some reason, there are no errors noted on either side either

Crypto Accelerator Status
-------------------------
[Capability]
   Supports hardware crypto: True
   Supports modular hardware crypto: False
   Max accelerators: 2
   Max crypto throughput: 750 Mbps
   Max crypto connections: 750
[Global Statistics]
   Number of active accelerators: 2
   Number of non-operational accelerators: 0
   Input packets: 840012
   Input bytes: 128403377
   Output packets: 1658183
   Output error packets: 0
   Output bytes: 1212761705

[Accelerator 0]
   Status: OK
   Software crypto engine
   Slot: 0
   Active time: 1051309 seconds
   Total crypto transforms: 48246
   Total dropped packets: 0
<--- More --->

   [Input statistics]
      Input packets: 0
      Input bytes: 118048
      Input hashed packets: 0
      Input hashed bytes: 0
      Decrypted packets: 0
      Decrypted bytes: 118048
   [Output statistics]
      Output packets: 0
      Output bad packets: 0
      Output bytes: 462400
      Output hashed packets: 0
      Output hashed bytes: 0
      Encrypted packets: 0
      Encrypted bytes: 462608
   [Diffie-Hellman statistics]
      Keys generated: 0
      Secret keys derived: 0
   [RSA statistics]
      Keys generated: 4
      Signatures: 12
      Verifications: 1
      Encrypted packets: 1
      Encrypted bytes: 28
<--- More --->

      Decrypted packets: 1
      Decrypted bytes: 256
   [ECDSA statistics]
      Keys generated: 12
      Signatures: 12
      Verifications: 15
   [SSL statistics]
      Outbound records: 0
      Inbound records: 0
   [RNG statistics]
      Random number requests: 300
      Random number request failures: 0
   [HMAC statistics]
      HMAC requests: 11438

[Accelerator 1]
   Status: OK
   Encryption hardware device : Cisco ASA-55xx on-board accelerator (revision 0x1)
                             Boot microcode        : CNPx-MC-BOOT-2.00
                             SSL/IKE microcode     : CNPx-MC-SSL-PLUS-T020
                             IPSec microcode       : CNPx-MC-IPSEC-MAIN-0026
   Slot: 1
   Active time: 1051314 seconds
   Total crypto transforms: 3032742
<--- More --->

   Total dropped packets: 0
   [Input statistics]
      Input packets: 840077
      Input bytes: 249823866
      Input hashed packets: 0
      Input hashed bytes: 0
      Decrypted packets: 840077
      Decrypted bytes: 128290297
   [Output statistics]
      Output packets: 1658289
      Output bad packets: 0
      Output bytes: 2316734504
      Output hashed packets: 377909
      Output hashed bytes: 89053304
      Encrypted packets: 1658294
      Encrypted bytes: 1191185759
   [Diffie-Hellman statistics]
      Keys generated: 225
      Secret keys derived: 65
   [RSA statistics]
      Keys generated: 0
      Signatures: 1
      Verifications: 1
      Encrypted packets: 2
<--- More --->

      Encrypted bytes: 63
      Decrypted packets: 2
      Decrypted bytes: 512
   [ECDSA statistics]
      Keys generated: 0
      Signatures: 0
      Verifications: 0
   [SSL statistics]
      Outbound records: 1280408
      Inbound records: 840095
   [RNG statistics]
      Random number requests: 4010
      Random number request failures: 0
   [HMAC statistics]
      HMAC requests: 106437

Not sure whats going on here but I have tried tearing stuff down, removing and reapplying cryptos, entire VPN configs removed and replaced, different transform sets, quadruple checked routing and nats and it all looks ok.

So to summarise, it looks like traffic is leaving the 5525x through the tunnel, arriving at the 5510, being successfully decrypted and turned around by the host (have proved this with a traffic capture on the 5510 LAN side), encrypted by the 5510, sent the the 5525x where it fails to be decrypted correctly. The same goes for traffic which originates from the LAN side of the 5510.