Symptoms
Cisco VPN Clients that connect to a VPN headend using IPSec over TCP might connect to the headend just fine, but then after some time the connection will fail. For this specific problem, switching to IPSec over UDP or native ESP IPSec encapsulation will work just fine.
Conditions
To encounter this specific problem, Cisco VPN Clients must be configured to connect to a VPN headend device using IPSec over TCP. Most commonly, we see network administrators configure the ASA to accept Cisco VPN Client connections over TCP Port 10000.
Problem
When the VPN client is configured for IPSec over TCP (known as cTCP) the VPN client software will not respond if a duplicate TCP ACK is received asking for the VPN client to re-transmit data. A duplicate ACK might be generated if there is packet loss somewhere between the VPN client and the ASA headend; intermittent packet loss is a fairly common reality on the internet. However, since the VPN endpoints are not using the TCP protocol (remember they're using cTCP), the endpoints will continue transmitting and the connection will continue.
A problem is only seen in this scenario if there is another device such as a firewall tracking the TCP connection statefully. Since the cTCP protocol does not fully implement a TCP client, and server duplicate ACKs will not be responded to, this can cause other devices in-line with this network stream to drop the TCP traffic. Packet loss must occur on the network causing TCP segments to go missing, which triggers the problem.
This isn't a bug, but a side effect of packet loss on the network, and the fact that cTCP is not real TCP. cTCP tries to emulate the TCP protocol by wrapping the IPSec packets within a TCP header, but that is the extent of the protocol.
This issue occurs usually when network administrators implement a ASA with an IPS, or do some sort of application inspection on the ASA that causes the firewall to act as a full TCP proxy of the connection. If there is packet loss, the ASA will ack for the missing data on behalf of the cTCP server or client, but the vpn client will never respond; since the ASA never receives the data it is expecting, communication cannot continue, causing the connection to fail.
Resolution
To resolve this problem, any of the following can be done:
- Switch from IPSec over TCP to IPSec over UDP, or native encapsulation with the ESP protocol.
- Switch to the AnyConnect client for VPN termination, which uses a fully implemented TCP protocol stack.
- Configure the ASA to apply tcp-state-bypass for these specific IPSec/TCP flows. This essentially disables all security checks for the connections that match the tcp-state-bypass policy, but will allow the connections to work until another resolution from this list can be implemented. More information on this feature can be found in the configuration guide:
4. Identify the source of the packet loss, and take corrective action to prevent the IPSec/TCP packets from being dropped on the network. This is usually impossible or extremely difficult since the trigger to the issue is usually packet loss on the internet, and the drops cannot be prevented.
