DTLS-Tunnel not passing traffic when VPN is connected

erik.kneebone · ‎07-27-2015

Hello all,

Thank you for taking the time to read this. We are having users who are able to connect to our client AnyConnect VPN, but are unable to pass traffic. It appears to be random. We have had several users reboot, restart, and reinstall their clients, then it starts working for a day or two and stops again. The symptom I've been able to find is the DTLS-Tunnel is not passing traffic. When they attempt to access the established SSL tunnels, they work. Below is the info regarding our ASAs:

Cisco Adaptive Security Appliance Software Version 9.2(3)4
Device Manager Version 7.4(1)
Hardware:   ASA5515, 8192 MB RAM, CPU Clarkdale 3058 MHz, 1 CPU (4 cores)
            ASA: 4096 MB RAM, 1 CPU (1 core)
Licensed features for this platform:
Maximum Physical Interfaces       : Unlimited      perpetual
Maximum VLANs                     : 100            perpetual
Inside Hosts                      : Unlimited      perpetual
Failover                          : Active/Active perpetual
Encryption-DES                    : Enabled        perpetual
Encryption-3DES-AES               : Enabled        perpetual
Security Contexts                 : 2              perpetual
GTP/GPRS                          : Disabled       perpetual
AnyConnect Premium Peers          : 250            perpetual
AnyConnect Essentials             : 250            perpetual
Other VPN Peers                   : 250            perpetual
Total VPN Peers                   : 250            perpetual
Shared License                    : Disabled       perpetual
AnyConnect for Mobile             : Enabled        perpetual
AnyConnect for Cisco VPN Phone    : Disabled       perpetual
Advanced Endpoint Assessment      : Disabled       perpetual
UC Phone Proxy Sessions           : 2              perpetual
Total UC Proxy Sessions           : 2              perpetual
Botnet Traffic Filter             : Disabled       perpetual
Intercompany Media Engine         : Disabled       perpetual
IPS Module                        : Enabled        perpetual
Cluster                           : Enabled        perpetual
Cluster Members                   : 2              perpetual

And now two examples, one who is working, and one who is not working:

show vpn-sessiondb detail anyconnect filter name

Session Type: AnyConnect Detailed

Username     : user2            Index        : 1342
Assigned IP : 192.168.100.222        Public IP    :
Protocol     : AnyConnect-Parent SSL-Tunnel DTLS-Tunnel
License      : AnyConnect Premium
Encryption   : AnyConnect-Parent: (1)none SSL-Tunnel: (1)RC4 DTLS-Tunnel: (1)AES128
Hashing      : AnyConnect-Parent: (1)none SSL-Tunnel: (1)SHA1 DTLS-Tunnel: (1)SHA1
Bytes Tx     : 10892                  Bytes Rx     : 99862
Pkts Tx      : 8                      Pkts Rx      : 1253
Pkts Tx Drop : 0                      Pkts Rx Drop : 0
Group Policy : GP_HomeOffice          Tunnel Group : DefaultWEBVPNGroup
Login Time   : 08:36:36 CDT Mon Jul 27 2015
Duration     : 0h:10m:11s
Inactivity   : 0h:00m:00s
VLAN Mapping : N/A                    VLAN         : none
Audt Sess ID : 0aa801fe0053e00055b633e4
Security Grp : none

AnyConnect-Parent Tunnels: 1
SSL-Tunnel Tunnels: 1
DTLS-Tunnel Tunnels: 1

AnyConnect-Parent:
Tunnel ID    : 1342.1
Public IP    :
Encryption   : none                   Hashing      : none
TCP Src Port : 49297                  TCP Dst Port : 443
Auth Mode    : userPassword
Idle Time Out: 30 Minutes             Idle TO Left : 19 Minutes
Client OS    : win
Client OS Ver: 6.1.7601 Service Pack 1
Client Type : AnyConnect
Client Ver   : Cisco AnyConnect VPN Agent for Windows 3.1.07021
Bytes Tx     : 5446                   Bytes Rx     : 782
Pkts Tx      : 4                      Pkts Rx      : 1
Pkts Tx Drop : 0                      Pkts Rx Drop : 0

SSL-Tunnel:
Tunnel ID    : 1342.2
Assigned IP : 192.168.100.222        Public IP    :
Encryption   : RC4                    Hashing      : SHA1
Encapsulation: TLSv1.0                TCP Src Port : 49300
TCP Dst Port : 443                    Auth Mode    : userPassword
Idle Time Out: 30 Minutes             Idle TO Left : 22 Minutes
Client OS    : Windows
Client Type : SSL VPN Client
Client Ver   : Cisco AnyConnect VPN Agent for Windows 3.1.07021
Bytes Tx     : 5446                   Bytes Rx     : 240
Pkts Tx      : 4                      Pkts Rx      : 3
Pkts Tx Drop : 0                      Pkts Rx Drop : 0

DTLS-Tunnel:
Tunnel ID    : 1342.3
Assigned IP : 192.168.100.222        Public IP    :
Encryption   : AES128                 Hashing      : SHA1
Encapsulation: DTLSv1.0               UDP Src Port : 49317
UDP Dst Port : 443                    Auth Mode    : userPassword
Idle Time Out: 30 Minutes             Idle TO Left : 29 Minutes
Client OS    : Windows
Client Type : DTLS VPN Client
Client Ver   : Cisco AnyConnect VPN Agent for Windows 3.1.07021
Bytes Tx     : 0                      Bytes Rx     : 98840
Pkts Tx      : 0                      Pkts Rx      : 1249
Pkts Tx Drop : 0                      Pkts Rx Drop : 0

show vpn-sessiondb detail anyconnect filter name

Session Type: AnyConnect Detailed

Username     : user2                 Index        : 87
Assigned IP : 192.168.100.219        Public IP    :
Protocol     : AnyConnect-Parent SSL-Tunnel DTLS-Tunnel
License      : AnyConnect Premium
Encryption   : AnyConnect-Parent: (1)none SSL-Tunnel: (1)RC4 DTLS-Tunnel: (1)AES128
Hashing      : AnyConnect-Parent: (1)none SSL-Tunnel: (1)SHA1 DTLS-Tunnel: (1)SHA1
Bytes Tx     : 692041                 Bytes Rx     : 494611
Pkts Tx      : 1611                   Pkts Rx      : 1732
Pkts Tx Drop : 0                      Pkts Rx Drop : 0
Group Policy : GP_HomeOffice          Tunnel Group : DefaultWEBVPNGroup
Login Time   : 08:25:11 CDT Mon Jul 27 2015
Duration     : 0h:22m:08s
Inactivity   : 0h:00m:00s
VLAN Mapping : N/A                    VLAN         : none
Audt Sess ID : 0aa801fe0005700055b63137
Security Grp : none

AnyConnect-Parent Tunnels: 1
SSL-Tunnel Tunnels: 1
DTLS-Tunnel Tunnels: 1

AnyConnect-Parent:
Tunnel ID    : 87.1
Public IP    :
Encryption   : none                   Hashing      : none
TCP Src Port : 53758                  TCP Dst Port : 443
Auth Mode    : userPassword
Idle Time Out: 30 Minutes             Idle TO Left : 16 Minutes
Client OS    : win
Client OS Ver: 6.1.7601 Service Pack 1
Client Type : AnyConnect
Client Ver   : Cisco AnyConnect VPN Agent for Windows 3.1.05160
Bytes Tx     : 1120                   Bytes Rx     : 776
Pkts Tx      : 1                      Pkts Rx      : 1
Pkts Tx Drop : 0                      Pkts Rx Drop : 0

SSL-Tunnel:
Tunnel ID    : 87.4
Assigned IP : 192.168.100.219        Public IP    :
Encryption   : RC4                    Hashing      : SHA1
Encapsulation: TLSv1.0                TCP Src Port : 57662
TCP Dst Port : 443                    Auth Mode    : userPassword
Idle Time Out: 30 Minutes             Idle TO Left : 16 Minutes
Client OS    : Windows
Client Type : SSL VPN Client
Client Ver   : Cisco AnyConnect VPN Agent for Windows 3.1.05160
Bytes Tx     : 1120                   Bytes Rx     : 0
Pkts Tx      : 1                      Pkts Rx      : 0
Pkts Tx Drop : 0                      Pkts Rx Drop : 0

DTLS-Tunnel:
Tunnel ID    : 87.5
Assigned IP : 192.168.100.219        Public IP    :
Encryption   : AES128                 Hashing      : SHA1
Encapsulation: DTLSv1.0               UDP Src Port : 55146
UDP Dst Port : 443                    Auth Mode    : userPassword
Idle Time Out: 30 Minutes             Idle TO Left : 29 Minutes
Client OS    : Windows
Client Type : DTLS VPN Client
Client Ver   : Cisco AnyConnect VPN Agent for Windows 3.1.05160
Bytes Tx     : 689801                 Bytes Rx     : 493835
Pkts Tx      : 1609                   Pkts Rx      : 1731
Pkts Tx Drop : 0                      Pkts Rx Drop : 0

When the one who has the failed DTLS packets, they are able to access the active IPsec tunnels:

show crypto ipsec sa | inc 192.168.100
      access-list L2L-AzureProd-VPN extended permit ip 192.168.100.0 255.255.255.0 172.17.0.0 255.255.0.0
      local ident (addr/mask/prot/port): (192.168.100.0/255.255.255.0/0/0)
      access-list outside_cryptomap_10 extended permit ip 192.168.100.0 255.255.255.0 10.158.11.0 255.255.255.128
      local ident (addr/mask/prot/port): (192.168.100.0/255.255.255.0/0/0)
      access-list outside_cryptomap_16 extended permit ip 192.168.100.0 255.255.255.0 10.158.14.0 255.255.255.128
      local ident (addr/mask/prot/port): (192.168.100.0/255.255.255.0/0/0)
      access-list outside_cryptomap_16 extended permit ip 192.168.100.0 255.255.255.0 10.158.14.128 255.255.255.128
      local ident (addr/mask/prot/port): (192.168.100.0/255.255.255.0/0/0)
      access-list L2L-Azure-VPN extended permit ip 192.168.100.0 255.255.255.0 172.16.0.0 255.255.0.0
      local ident (addr/mask/prot/port): (192.168.100.0/255.255.255.0/0/0)
      access-list outside_cryptomap_13 extended permit ip 192.168.100.0 255.255.255.0 10.158.129.0 255.255.255.128
      local ident (addr/mask/prot/port): (192.168.100.0/255.255.255.0/0/0)
      access-list outside_cryptomap_110 extended permit ip 192.168.100.0 255.255.255.0 10.158.3.0 255.255.255.128
      local ident (addr/mask/prot/port): (192.168.100.0/255.255.255.0/0/0)
      access-list outside_cryptomap_3 extended permit ip 192.168.100.0 255.255.255.0 10.158.1.0 255.255.255.128
      local ident (addr/mask/prot/port): (192.168.100.0/255.255.255.0/0/0)

But nothing beyond that.

I'm looking for any troubleshooting steps which could help me track down the problem or provide suggestions if possible. It seems a fix for this may be to disable DTLS. I have users with the following connection who have not complained:

MTM-MO-FW1# show vpn-sessiondb detail anyconnect filter name

Session Type: AnyConnect Detailed

Username     : user3               Index        : 822
Assigned IP : 192.168.100.247        Public IP    :
Protocol     : AnyConnect-Parent SSL-Tunnel
License      : AnyConnect Premium
Encryption   : AnyConnect-Parent: (1)none SSL-Tunnel: (1)RC4
Hashing      : AnyConnect-Parent: (1)none SSL-Tunnel: (1)SHA1
Bytes Tx     : 7650336                Bytes Rx     : 2517980
Pkts Tx      : 17185                  Pkts Rx      : 13163
Pkts Tx Drop : 0                      Pkts Rx Drop : 0
Group Policy : GP_HomeOffice          Tunnel Group : DefaultWEBVPNGroup
Login Time   : 06:09:29 CDT Fri Jul 24 2015
Duration     : 3d 2h:53m:37s
Inactivity   : 0h:00m:00s
VLAN Mapping : N/A                    VLAN         : none
Audt Sess ID : 0aa801fe0033600055b21ce9
Security Grp : none

AnyConnect-Parent Tunnels: 1
SSL-Tunnel Tunnels: 1

AnyConnect-Parent:
Tunnel ID    : 822.1
Public IP    :
Encryption   : none                   Hashing      : none
TCP Src Port : 52794                  TCP Dst Port : 443
Auth Mode    : userPassword
Idle Time Out: 30 Minutes             Idle TO Left : 1 Minutes
Client OS    : win
Client OS Ver: 6.1.7601 Service Pack 1
Client Type : AnyConnect
Client Ver   : Cisco AnyConnect VPN Agent for Windows 3.1.07021
Bytes Tx     : 4287001                Bytes Rx     : 2588
Pkts Tx      : 4186                   Pkts Rx      : 6
Pkts Tx Drop : 0                      Pkts Rx Drop : 0

SSL-Tunnel:
Tunnel ID    : 822.5
Assigned IP : 192.168.100.247        Public IP    :
Encryption   : RC4                    Hashing      : SHA1
Encapsulation: TLSv1.0                TCP Src Port : 57295
TCP Dst Port : 443                    Auth Mode    : userPassword
Idle Time Out: 30 Minutes             Idle TO Left : 30 Minutes
Client OS    : Windows
Client Type : SSL VPN Client
Client Ver   : Cisco AnyConnect VPN Agent for Windows 3.1.07021
Bytes Tx     : 3363706                Bytes Rx     : 2515518
Pkts Tx      : 13001                  Pkts Rx      : 13159
Pkts Tx Drop : 0                      Pkts Rx Drop : 0

taylorML1 · ‎08-07-2015

Hi Erik,

I have to say I am dealing with something similar and it has been a very frustrating process to attempt to nail this down. but I'd like to compare notes with you to see if we are having the same problems.

I have three users that use AnyConnect to connect to one of our Customers networks, That customer has given me limited information on how their ASA is configured, but keep telling me that the problem is "in my network" and not in their ASA. While I'm running out of ways to tell them that the other 1900 users on 'my network' have no problems with VPN or AnyConnect, and honestly its best if I help get it working so we can keep billing them.

My scenario is Three users connected to a Cisco 3560x switch, and Running through a Cisco 2951 router. I'm running NVI nat on the router and basic fire-walling.

The Users will be working along and suddenly stop passing traffic through the tunnel. The AnyConnect Client does not say "reconnecting" it does not drop the tunnel, It just simply increments the "sent" counter, and nothing increments on the "received" counter. Then after 30-45 seconds the traffic picks back up and the users can re-connect their outlook clients and mapped drives.

Sadly I can't get to the ASA to look at logs or see both sides of the traffic when its happening so I'm focusing on what I can see and do.

So a packet capture showed that at about the time of the incident the source ports changed indicating a new connection request that is outside the previous connection stream.

While looking at the router I noticed that there is a very, VERY strange address translation that pops into the translation table, and I can't for the life of me figure out how it got there and or why.

In short it looks to be a mirror connection of the new session request for DTLS.

Here is a basic breakdown of what I'm describing.

--------------working section ------ Notice one TCP connection and One UDP connection (tls control and DTLS data)

router-name-was-here#show ip nat nvi trans vrf VRFNAMEWASHERE | inc 192.168.76.51

udp pub.ip.was.here:49180 192.168.76.51:49180 dst.ip.was.here:443 dst.ip.was.here:443

tcp pub.ip.was.here:58077 192.168.76.51:58077 dst.ip.was.here:443 dst.ip.was.here:443

router-name-was-here#show ip nat nvi trans vrf VRFNAMEWASHERE | inc 192.168.76.51

udp pub.ip.was.here:49180 192.168.76.51:49180 dst.ip.was.here:443 dst.ip.was.here:443

tcp pub.ip.was.here:58077 192.168.76.51:58077 dst.ip.was.here:443 dst.ip.was.here:443

Now to Switch to a section where the issue is happening or has happened. Notice what appears to be a mirror of the newly created NAT entry using my public IP and port 443 and their destination IP as the source port 62758. Also note the old UDP translation is still active. So that appears to be a valid connection still. while the in this state the connection is dead. No traffic will pass on the VPN but the client still says "connected"

router-name-was-here#show ip nat nvi trans vrf VRFNAMEWASHERE | inc 192.168.76.51

udp pub.ip.was.here:443 192.168.76.51:443 dst.ip.was.here:62758 dst.ip.was.here:62758

udp pub.ip.was.here:49180 192.168.76.51:49180 dst.ip.was.here:443 dst.ip.was.here:443

tcp pub.ip.was.here:58077 192.168.76.51:58077 dst.ip.was.here:443 dst.ip.was.here:443

udp pub.ip.was.here:62758 192.168.76.51:62758 dst.ip.was.here:443 dst.ip.was.here:443

And lastly when it begins working again. Notice the Old Entry has timed out and now we are left with the one "mirror" entry that appears to me to be completely invalid, and then the normal tcp and udp connections.

router-name-was-here#show ip nat nvi trans vrf VRFNAMEWASHERE | inc 192.168.76.51

udp pub.ip.was.here:443 192.168.76.51:443 dst.ip.was.here:62758 dst.ip.was.here:62758

tcp pub.ip.was.here:58077 192.168.76.51:58077 dst.ip.was.here:443 dst.ip.was.here:443

udp pub.ip.was.here:62758 192.168.76.51:62758 dst.ip.was.here:443 dst.ip.was.here:443

It will then run in that sometime for a long time, sometimes for 10 minutes..before it happens again.

My packet captures look like when the issue starts the connection appears to drop the DTLS connection and begin using only TLS, Like they have enabled both on the AnyConnect group on their ASA, and the Client is giving up on DTLS and reverting. I can't however see inside the packets to know what they are trying to do, or why since its all encrypted and I can't get inside the laptop.

I don't know if this sounds familiar to your situation or if this is completely different but I've been running into dead ends with every avenue i take to determine what would cause these strange NAT entries, and why the VPN tunnel would remain "connected" but stop passing traffic properly.

erik.kneebone · ‎08-10-2015

Due to the high impact of this problem we upgraded to 9.4(1) which is presenting it's own problems as well, but they are at least manageable. We attempted multiple versions of the AnyConnect client and no luck. It seemed there was no reliable combo. We have 100+ remote workers on any given day so we could not sit and wait unfortunately. 9.4(1) appears to have some VPN memory leak issues where the Unicorn processes start maxing out the CPU process after a few days. Fortunately we are able to fail over to our standby unit and reload.

I wish I had a better answer.