This document provides the basic procedures for identifying, understanding, and mitigating asymmetric routing issues in networks that are protected by the Cisco Adaptive Security Appliance (ASA).
The ASA protects the network by denying traffic unless it can inspect both sides of the flow. To allow the ASA to continue maintaining the security of the network, the correct and ideal solution for asymmetric routing is to resolve the problem that causes the asymmetry elsewhere in the network. However, this document is designed to help identify the problem and provide workarounds until the asymmetry can be fixed.
The term asymmetric routing refers to a packet or connection flow that takes different paths through the network in the forward and reverse directions. For example, a packet may leave the internal network destined for the Internet through ASA1, but the server's response to that packet may return from the Internet and reach the network through ASA2.
Because the ASA keeps track of the active connections passing through the firewall, packets that arrive and that are not part of an existing connection or explicitly permitted by an access list will be dropped when asymmetric routing occurs. In the example from the previous paragraph, the first packet leaving for the Internet through ASA1 will cause ASA1 to build an entry in its connection table. Once this entry is built, all return packets from the Internet that are specific to this connection will be allowed through until the connection is removed from the table. Since ASA2 never saw the 1st packet, no connection is built and thus it is not expecting the response packet from the Internet, which causes the packet to be dropped without an explicit rule to allow it. The following sections in this document will provide alternatives to prevent these packets from being dropped by the ASA.
Example A - Normal Flow
In order to understand what asymmetric routing looks like from a diagnostic point of view, it is important to first understand how a typical packet flow will look. Consider the following simple topology:
When the client wants to send HTTP traffic to the web server at www.cisco.com, it will start a TCP connection by sending its first packet to its default gateway, which is the ASA. The ASA will then route the packet out to the Internet so that it can eventually reach the web server. When the web server receives the request, it will send its response back toward the client, which will eventually arrive at the ASA. Since, having seen the initial request, the ASA knows about this connection, it will allow the response through and forward it back to the client.
Using the packet capture feature of the ASA, it is easy to see that the routing in this topology is symmetric. In the example below, packet #1 arrives on the inside interface of the ASA and leaves on the outside interface of the ASA. Likewise, the response packet, packet #2, returns on the outside interface and is forwarded on through the inside interface. Seeing bidirectional communication on both of the ASA's interfaces is a good sign that everything is working as expected.
One common example of the asymmetric routing problem occurs when a network has more than one path to the Internet. Depending on the way routing is configured at the network edge or by the ISP, the initial packet may leave through one path but the response returns through a different path as shown in the example below.
10:45:58.734881 IP 192.168.1.1.58458 > 220.127.116.11.80: S 2331481362:2331481362(0) win 5840 <mss 1460,sackOK,timestamp 2359730964 0,nop,wscale 7> 10:45:51.787530 IP 18.104.22.168.80 > 192.168.1.1.58458: S 712296788:712296788(0) ack 126872521 win 16384 <mss 1380>
In this example, the client sends packet #1 to its default gateway, which is the ASA. It is routed through the Internet where it reaches the web server. When the web server responds, routing takes the reply back through the Internet where it arrives on RouterA, who forwards it to RouterB, who sends it directly to the client. Note that because the ASA randomizes the TCP sequence and acknowledgement numbers by default, the server will respond with a SYN ACK packet that has an ACK value of 126872521 (the andomized initial sequence number from the SYN packet + 1). Because the sequence and acknowledgement numbers are now out of sync, the client will not be able to establish the connection because the client expects the ACK value to be 2331481363.
Example C - Active/Active Failover
Another issue with asymmetric routing can arise when 2 ASAs are configured in Active/Active failover mode.
In this example, ASA1 and ASA2 are connected together in an Active/Active failover mode. There are 2 contexts configured on each unit, ContextA and ContextB. ContextA is active on ASA1 and standby on ASA2, while ContextB is active on ASA2 and standby on ASA1.
When a packet leaves the client destined for www.cisco.com, it arrives at its default gateway of the inside interface of ContextA. ContextA then forwards it on toward the Internet where it reaches the server. Then, the server sends its reply back toward the client. When the response arrives at the Router, it is routed to the outside interface of ContextB, rather than ContextA where the original request came from. Because ContextB never saw the initial request and does not have an existing entry its connection table, the packet is dropped and thus the client is not able to establish a connection with the server. Packet captures on the firewall contexts will clearly illustrate the asymmetry, while the client will simply see no response from the server.
11:25:18.201065 IP 192.168.1.1.40400 > 22.214.171.124.80: S 518915629:518915629(0) win 5840 <mss 1460,sackOK,timestamp 2361776640 0,nop,wscale 7> 11:25:18.201065 IP 192.168.1.1.40400 > 126.96.36.199.80: S 518915629:518915629(0) win 5840 <mss 1460,sackOK,timestamp 2361776640 0,nop,wscale 7>
Packet captures on the ASA can help easily identify asymmetric routing issues. For example, if you see only one-way traffic (i.e. only client to server traffic), asymmetric routing may be present.
It is important to setup the captures on the ASA to be both bidirectional (i.e. client to server, and server to client) and simultaneous on both sides of the firewall (i.e. both the inside and outside interfaces). For more information on using the ASA's packet capture utility, please see the following document:
Syslogs are another useful indication of asymmetric routing. For example, if asymmetric routing exists and the ASA only sees one side of a given connection, a successful connection will never be established and thus many packets will be denied for this reason. In this example, the ASA will log the following message at the informational level:
%ASA-6-106015: Deny TCP (no connection) from 188.8.131.52/80 to 192.168.1.1/40400 flags SYN ACK on interface outside
The following sections outline the available mitigation techniques when the ASA drops packets due to asymmetric routing.
One of the best, and perhaps most obvious, solutions is to simply fix the routing configuration in the packet's path so that the ASA sees both sides of the connection. In Example B above, RouterA's routing table should send all packets destined for the client to the ASA's outside interface, rather than sending them to RouterB.
Unfortunately, this solution is not always available to network administrators. In some cases, the asymmetry could be caused by devices outside the administrator's control, such as routers managed by an ISP or 3rd party vendor. When this occurs, it is still usually best to contact the responsible organization and work with them to resolve the issue, but the remaining mitigation techniques on the ASA may also be used.
For situations similar to Example C above, ASA's that are configured for Active/Active Stateful Failover can put their interfaces into ASR groups:
ASA/ContextA(config)# interface Vlan 101
ASA/ContextA(config-if)# nameif outside
ASA/ContextA(config-if)# asr-group 1
ASA/ContextB(config)# interface Vlan201
ASA/ContextB(config-if)# nameif outside
ASA/ContextB(config-if)# asr-group 1
When an interface that is not in an ASR group receives a packet that it does not have an established connection for, it will normally drop it. However, when an interface is in an ASR group, the ASA checks the connection information for all other interfaces in the same group. If the ASA finds another context that has connection information that would match the received packet, the Layer 2 header information is rewritten and the packet is redirected to the appropriate context for processing.
TCP State Bypass
Specifically for TCP-based connections, disabling stateful TCP checks can help mitigate asymmetric routing. When TCP state checks are disabled, the ASA can allow packets in a TCP connection even if the ASA didn't see the entire TCP 3-way handshake. This feature is called TCP State Bypass (introduced in ASA 8.2).
ASA(config)# access-list tcp_bypass extended permit tcp 192.168.1.0 255.255.255.0 any ASA(config)# class-map tcp_bypass ASA(config-cmap)# match access-list tcp_bypass ASA(config-cmap)# policy-map tcp_bypass_policy ASA(config-pmap)# class tcp_bypass ASA(config-pmap-c)# set connection advanced-options tcp-state-bypass ASA(config-pmap-c)# set connection timeout idle 0:10:00 ASA(config-pmap-c)# service-policy tcp_bypass_policy inside
For ASA's running software versions prior to 8.2, similar functionality can be enabled using the 'norandomseq nailed' option at the end of a static statement. For more information on this configuration, see the following document:
Please note that using TCP State Bypass (or the 'nailed' option) will also disable all TCP-based security checks and application inspection. Also, until the fix for enhancement CSCtk84429 is implemented, it is recommended that when TCP State Bypass is enabled an explicitly lower idle timeout is configured to prevent the ASA from reaching its resource limit in high connection environments. This is because the ASA no longer removes a TCP connection from its table based on a TCP FIN teardown (since TCP state checking no longer occurs). As a result, the ASA will have to wait until the global idle timer expires (1 hour by default) before the connection is removed.
The ASA protects the network by blocking traffic flows unless it is able to inspect the entire (both sides) of the connection. Using tools such as packet captures and syslogs, the ASA can also be a useful troubleshooting tool in identifying asymmetric routing problems. Although the ASA keeps the network secure by blocking this traffic, the above workarounds can be used to allow traffic through the firewall until the problem can be solved permanently.
Configuration Guide - Configuring Support for Asymmetrically Routed Packets: