Cisco Support Community
cancel
Showing results for 
Search instead for 
Did you mean: 

ASA NAT Migration problems when upgrading to 8.3; Syslog "%ASA-5-305013: Asymmetric NAT rules matched for forward and reverse flows"

Note: To see a short video explaining the basics of the new 8.3 and 8.4 NAT policy, see the link below:

video-screenshot.pngVIDEO: Cisco ASA version 8.3 and 8.4 NAT Configuration Example

https://supportforums.cisco.com/docs/DOC-12324

Scope of this document:

This document describes a situation that might be encountered when upgrading an ASA from version 8.1 or 8.2 to version 8.3. In particular, configurations involving VPN connections and "nat 0" statements might encounter this issue. The problem is not a bug, but a side effect of how Network Address Translation (NAT) works in version 8.3.

One of the symptoms of this problem is that some traffic will be dropped by the ASA, and the log

%ASA-5-305013: Asymmetric NAT rules matched for forward and reverse flows

will be generated by the firewall.

Notice

Upgrading to version 8.3(2) might add the 'unidirectional' keyword to all policy-nat exemption lines due to the bug referenced below. This problem causes the same symptoms as the issue discussed in the remainder of this document:


CSCti36048
    ASA upgrade to 8.3(2) adds unidirectional keyword to manual nat lines


This issue most commonly causes a problem with remote VPN subnets attempting to send traffic inbound to the ASA; after the upgrade this traffic might fail. Removing the 'unidirectional' keyword will allow inbound connections from the VPN client to match these manual nat translations, and the connection should work ok (as per the bug release-note). Please continue reading to learn about other potential (non-bug) causes for the Asymmetric NAT rule problem

Topology

Screen shot 2010-08-12 at 12.37.41 PM.png

Consider the above toplogy, showing an ASA with three interfaces; inside, outside, and dmz. The version 8.2 NAT configuration has the following statements that perform nat on the traffic:

access-list nonatinside extended permit ip 10.10.0.0 255.255.0.0 192.168.1.0 255.255.255.0 
access-list nonatdmz extended permit ip 10.10.6.0 255.255.255.0 192.168.1.0 255.255.255.0

nat (inside) 0 access-list nonatinside
nat (dmz) 0 access-list nonatdmz
nat (inside) 1 0.0.0.0 0.0.0.0
global (outside) 1 interface
static (dmz,Outside) 14.36.101.74 10.10.6.20 netmask 255.255.255.255


Explanation of version 8.2 NAT configuration:

access-list nonatinside extended permit ip 10.10.0.0 255.255.0.0 192.168.1.0 255.255.255.0 
nat (inside) 0 access-list nonatinside

This policy nat-exemption statement dictates that traffic sourced from the 10.10.0.0/16 network on the inside interface should not be translated when accessing the 192.168.1.0/24 network across the tunnel on the outside interface.


nat (dmz) 0 access-list nonatdmz

Similar to the first statement, this dictates that traffic received on the dmz interface sourced from the 10.10.6.0/24 network destined to the remote 192.168.1.0/24 network should not be translated by the ASA.



nat (inside) 1 0.0.0.0 0.0.0.0
global (outside) 1 interface

These lines provide outbound Port Address Translation for traffic flowing from the inside to the outside interface.



static (dmz,Outside) 209.165.201.5 10.10.6.20 netmask 255.255.255.255

This is a static translation mapping the local address of 10.10.6.20 to the global 209.165.201.5. This allows outside users to connect directly to the ip 209.165.201.5 and connect to the dmz server. When this ASA is upgraded to version 8.3(1), the NAT translation will be migrated to the new NAT configuration style. This new NAT scheme is completely object-oriented and the syntax from the previous NAT configuration is different. More information on the new NAT system can be found here:

VIDEO: Cisco ASA version 8.3 NAT Configuration Example

ASA 8.3 Command Line Configuration Guide; Configuring NAT

ASA Pre-8.3 to 8.3 NAT configuration examples:

After the migration, the complete converted NAT config will look like this:


object network obj-10.10.0.0
subnet 10.10.0.0 255.255.0.0
object network obj-192.168.1.0
subnet 192.168.1.0 255.255.255.0
object network obj-10.10.6.0
subnet 10.10.6.0 255.255.255.0
object network obj-10.10.6.20
host 10.10.6.20
object network obj_any
subnet 0.0.0.0 0.0.0.0
!
nat (inside,any) source static obj-10.10.0.0 obj-10.10.0.0 destination static obj-192.168.1.0 obj-192.168.1.0
nat (dmz,Outside) source static obj-10.10.6.0 obj-10.10.6.0 destination static obj-192.168.1.0 obj-192.168.1.0
!
object network obj_any
nat (inside,Outside) dynamic interface
object network obj-10.10.6.20
nat (dmz,Outside) static 209.165.201.5

However, after the upgrade inbound connections from the remote host at 192.168.1.5 destined to the DMZ server IP 10.10.6.20 will fail; Inbound ping packets received over the tunnel that match this flow will be dropped by the ASA, and the following syslog will be generated:


%ASA-5-305013: Asymmetric NAT rules matched for forward and reverse flows;
Connection for icmp src Outside:192.168.1.5 dst dmz:10.10.6.20 (type 8, code 0) 
denied due to NAT reverse path failure

The reason for this is that an overlap exists in the manual nat statements. To see exactly how this is happening, we can issue 'show nat detail' to see more information about the NAT table on the ASA:



ASA# show nat
Manual NAT Policies (Section 1)
1 (inside) to (any) source static obj-10.10.0.0 obj-10.10.0.0 destination static obj-192.168.1.0 obj-192.168.1.0 unidirectional
    translate_hits = 0, untranslate_hits = 0
2 (dmz) to (Outside) source static obj-10.10.6.0 obj-10.10.6.0 destination static obj-192.168.1.0 obj-192.168.1.0 unidirectional
    translate_hits = 0, untranslate_hits = 0

Auto NAT Policies (Section 2)
1 (dmz) to (Outside) source static obj-10.10.6.20 209.165.201.5
    translate_hits = 0, untranslate_hits = 0
2 (inside) to (Outside) source dynamic obj_any interface
    translate_hits = 0, untranslate_hits = 0

In this case, the problem stems from the first entry in the manual NAT table. In version 8.3, extra checks are performed on packets traversing the firewall if they are subjected to address translation. Specifically, the ASA checks to ensure that a local-ip (a real host ip) exists only on one interface, so that there is no possibility of an address translation overlap. When it detects this problem, it logs the syslog 305013. In this case, the problem is caused by the first manual NAT statement. The configuration lines below:


object network obj-10.10.0.0
subnet 10.10.0.0 255.255.0.0

object network obj-192.168.1.0
subnet 192.168.1.0 255.255.255.0
nat (inside,any) source static obj-10.10.0.0 obj-10.10.0.0 destination static obj-192.168.1.0 obj-192.168.1.0

Notice that the host we are trying to ping on the DMZ is 10.10.6.20, yet the first manual NAT statement (which references the object obj-1010.0.0) indicates that the object "obj-10.10.0.0" represents the real ip addresses of hosts behind the inside interface; obj-10.10.0.0 is defined as the 10.10.0.0/16 network, and this subnet overlaps with the subnet that resides behind the DMZ interface. Because of this overlap, the packet fails the NAT check, and the inbound packet is denied by the firewall.

The solution to the problem is to refine the local object in the first manual nat statement, so that the subnet does not overlap with the subnet on the dmz. Here we create a new object called 'inside-network' which is more specific, and does not overlap with the dmz. Then we remove the old manual NAT statement, and replace it with a new one which references the new object we created


object network inside-network
subnet 10.10.5.0 255.255.255.0

nat (inside,any) source static inside-network inside-network destination static obj-192.168.1.0 obj-192.168.1.0

So that the complete, working NAT configuration looks like this:


object network obj-192.168.1.0
subnet 192.168.1.0 255.255.255.0
object network obj-10.10.6.0
subnet 10.10.6.0 255.255.255.0
object network obj_any
subnet 0.0.0.0 0.0.0.0
object network inside-network
subnet 10.10.5.0 255.255.255.0
object network obj-10.10.6.20
host 10.10.6.20

!
nat (inside,any) source static inside-network inside-network destination static obj-192.168.1.0 obj-192.168.1.0
nat (dmz,Outside) source static obj-10.10.6.0 obj-10.10.6.0 destination static obj-192.168.1.0 obj-192.168.1.0
!
object network obj_any
nat (inside,Outside) dynamic interface
object network obj-10.10.6.20
nat (dmz,Outside) static 209.165.201.5

After doing this, the traffic no longer fails the Asymmetric check, and is passed through to the DMZ. Here is a syslog showing the traffic passing through the ASA:


%ASA-6-302020: Built inbound ICMP connection for faddr 192.168.1.5/9481
gaddr 10.10.6.20/0 laddr 10.10.6.20/0

ASA upgrade to version 8.3(2) adds 'unidirectional' keyword to manual NAT lines

Please see the following bug:

Version history
Revision #:
1 of 1
Last update:
‎08-12-2010 12:30 PM
Updated by:
 
Labels (1)
Comments
New Member

Jay,

How can I thank you for this article?  I unfortunately upgraded a critical production network on Friday the 13th and spent most of the night and this morning trying to resolve VPN connectivity issues.  We worked with 3 different TAC engineers and apparently they were not aware of this issue.  After reading you article I had the network services restored in 5 minutes.  Thanks for saving my @ss!

New Member

Thanks!  This article saved me a TON of time when I installed my first 8.3 ASA this week, previously I'd been installing 8.0 ASAs with no problems but I thought, yeah what the heck lets try the newest firmware and then I started beating my head against the desk when it didn't work like previous installs.  But your article saved the day for me!  Thanks.

Thomas Dzubin

New Member

I've tried following this guide but I'm still having trouble no-natting VPN clients per https://supportforums.cisco.com/message/3168125

New Member

Hi,

why you have do the following config

object network obj-10.10.6.20
nat (dmz,Outside) static 209.165.201.5

and not this?

object network obj-10.10.6.20
host 10.10.6.20

nat (dmz,Outside) static 209.165.201.5

thanks a lot best regards

New Member

Jay,

I would laos like to relay my thanks for this article.

When upgrading to 8.3 at my previous place of employment i didn't experience any issues with the VPN configuration. But over the last week after upgrading form 8.2 to 8.3 and creating the Remote Access VPN connection  i experienced the smae issue as Steve Dussault. Very helpful and concise descreption of the issue.

Cheers

Frankie John-Lewis

New Member

I recently did another upgrade and the access list for the outside interface did not get updated with the internal IP address of the outside objects.  Cisco needs to publish a tool that you can run the 8.2 configs through prior to the upgrade to identify what the automatic upgrade (???) will break.

Cisco Employee

Frankie, thanks for the feedback, I'm glad the document helped!

Cisco Employee

Steve,

Check to see if you hit this bug:

CSCtf57830 Incorrect real ip translation of ACE after 8.3.1 upgrade

at www.cisco.com/go/bugs

The trigger is a "nat (interface) 0 access-list" where the access-list has the keyword 'any' in it.

New Member

Hi Jay - This doc is awesome, but I have a few questions:

What if you have multiple intenal networks, can you use objet groups in the NAT rules

Also, as a previous reader commented, why have you not used the host statement for the the object:

object network obj-10.10.6.20

host 10.1.6.20?

and the last question, how would you read the following the nat statement, is it

nat (dmz,outside) static 209.165.201.9

When the traffic is sourced "from" the dmz interface "to" the outside interface, translate the source as 209.165.201.9?  Does this statement NAT the destination server when the traffic is ourced from the remote access client. For example, if my anyconnect pool is 192.168.100.0/24 and i get asigned 192.168.100.10 and I access this server in the DMZ, can I access it using its internal IP?

Thanks again for your doc - its cleared many other questions I had..

Cisco Employee

Zahir,

Yes, one can use object-groups in the nat rules. Below is an example, where I have multiple global IP addresses available for dynamic one-to-one NAT and a single PAT IPaddress for fallback. While not exactly the same situation you describe, it illustrates the behavior:

Pre 8.3 Configuration:

nat (inside) 1 10.10.0.0 255.255.0.0
global (outside) 1 209.165.201.1-209.165.201.30
global (outside) 1 209.165.201.31
global (outside) 1 209.165.201.32

8.3 and later configuration:

object network MAPPEDip1
host 209.165.201.31

object network MAPPEDip2
host 209.165.201.32

object network MAPPEDrange
range 209.165.201.1 209.165.201.30

object-group network MappedObjectGrp
network-object object MAPPEDrange
network-object object MAPPEDip1
network-object object MAPPEDip2
!

nat (inside,outside) source dynamic obj-10.10.0.0 MappedObjectGrp (corrected)

nat (dmz,outside) source dynamic obj-10.10.0.0 MappedObjectGrp

Regarding your second question, you're right I missed including the object definition for 10.10.6.20...I'll add that to the example. Thanks!

You are correct with your last point, this statement:

nat (dmz,outside) static 209.165.201.9

Can be read as: "When this host sends traffic outbound from the dmz to the outside interface, translate its source to 209.165.201.9. Likewise, when any host on the outside sends an IP packet destined to 209.165.201.9, translate the destination to the real IP and forward to the inside interface".

The hosts on the outside should connect to the global IP of 209.165.201.9, and not the mapped ip. You could allow your vpn users on the outside to connect to the global by creating a manual nat rule higher in the nat table, specifying the external subnet, like this:

nat (inside,outside) static dmz-server dmz-server destination static vpn-users vpn-users

Then, when remote users tried to access the local IP of the server (dmz-server) they would hit this rule and go through the firewall untranslated, since they didn't match the translation rule farther down the table.

New Member

Jay,

first of all thank you for writing this article. It has been very helpful in understanding the 8.3 NAT.

Also, in your response to Zahir, last NAT statement in relation to vpn users accessing DMZ server via its real-local-ip address, you wrote

"nat (inside,outside) static dmz-server  dmz-server destination static vpn-users vpn-users" .

Didn't you actually mean to write "nat (dmz,outside) static dmz-server  dmz-server destination static vpn-users vpn-users" .

Does the same principle apply for VPN networks coming across via a site-to-site VPN tunnel ?

Do we need to arrange or configure NAT rules in v8.3 in a particular order ?

New Member

Hi Jay - Thank you  very much for taking your time and replying to my questions

If it was not for your videos and docs on Cisco support community i think I would have pulled out the little hair that I have left  :-)

I have done a very similar thing to what you recomended and placed  rules in section 1 so that they are executed before the other sections as follows:

nat (inside,any) source static INSIDE_NETS INSIDE_NETS destination static SSLPOOL SSLPOOL

where INSIDE_NETS is my object group for all my internal subnets and the SSLPOOL is the anyconnect client pool.

A similar statement was also added to allow the DMZ servers to "no NAT" when accessed from anyconnect clients as follows:

nat (dmz1,outside) source static DMZ1-172.18.0.0 DMZ1-172.18.0.0 destination static SSLPOOL SSLPOOL

Since I have two DMZ subnets another rule was added similar to the one above. Then in section 2 of the NAT table I have all my statics configured under the objects as follows:

object network OBJ-ANY
nat (inside,outside) dynamic interface    (This does PAT for all internet access)

object network S-UNI-DMC01
nat (inside,outside) static x.xx.x (mapped IP)  (This is one of many DMZ servers accessible from the internet)

Once again Jay - THANK YOU SO MUCH!!   and keep the videos coming please..

New Member

Hi Zahir,

Could you kindly explain me how you are configuring the NATs as per sections ? Is this through the ASDM ?

The reason I ask is when Twice NAT and Network Object NAT are configured via the CLI, the show run nat always puts the Twice NAT on the top followed by Network Object NAT.

So I'm a bit confused as to how you are putting it into different sections and secondly does it have any significance ?

Thanks

New Member

Hi Vivek,

When you do the show nat command you will see that the rules appear in different sections. You are right, when you configure the  twice NAT rules they appear at the top automatically; when you configure an object and configure NAT for it under the object, they appear in section 2 of the NAT table. You can do it via both the ASDM and the CLI. As I mainly use the CLI, there is an option called "after-auto" when configuring NAT rules and these are then placed in section 2. The ASDM also gives you the option to create the rules in the various sections. The NAT rules are executed in order, so twice NAT rules appear first, followed by "auto nat" rules and then section 3. Section 3  also holds twice nat rules and are usually used when your VPN does not work. You can specify when configuring twice NAT which section you want them to be in, section 1 or 3.

Hope this helps and here is the doc that holds this information:

http://www.cisco.com/en/US/docs/security/asa/asa83/configuration/guide/nat_overview.html#1122015

Cisco Employee

Yes, I meant to say "nat (dmz,outside)", I'll correct the above post

Does the same principle apply for VPN networks coming across via a site-to-site VPN tunnel ?

Answer: Yes, the NAT configuration applies to all traffic, regardless of if it is passing over a VPN tunnel or not. Often, administrators configure a NAT policy to NOT translate traffic going over the tunnel (since it is internal traffic and doesn't require translation).

Do we need to arrange or configure NAT rules in v8.3 in a particular order?

Answer: The order of the NAT rules does matter; the NAT configuration builds a set of rules that can be observed using the 'show nat' and 'show nat detail' command. The rules are applied in the order seen in this table, and the rule ordering can be modified