ASA 8.4(2) NAT Asymmetric NAT rules matched for forward and reverse flows

[Andrey Padiy]

Hi,

We are experiencing very intermittent issue with NATing. It happened twice now in two weeks. First time it lasted for about 10 minutes, second time I had to failover to satandby firewall.

We have Cisco ASA 5510 with 4 IPSec VPN tunnels. Everything works as expected but occasionally we get :

"Asymmetric NAT rules matched for forward and reverse flows..."

When this happenes NAT outside works but connectivity to VPN sites is lost.

IPSec tunnels remains established between all site.

Here is our NAT config

FW# show running-config nat

nat (inside,outside) source static OfficeLAN OfficeLAN destination static XXXXXX-XXXX  XXXXX-XXXX

nat (inside,outside) source static OfficeLAN OfficeLAN destination static XXXXXX-XXXX  XXXXX-XXXX

nat (inside,outside) source static OfficeLAN OfficeLAN destination static XXXXXXX-XXXX XXXXX-XXXX

nat (inside,outside) source static OfficeLAN OfficeLAN destination static XXXXXXX-XXXX XXXXX-XXXX

nat (inside,outside) source dynamic any interface

nat (inside,outside_backup) source dynamic any interface

nat (GuestLAN,outside) source static GuestLAN GuestLAN destination static XXXXXXX-XXXX XXXXX-XXXX

nat (GuestLAN,outside) source static GuestLAN GuestLAN destination static XXXXXXX-XXXX XXXXX-XXXX

nat (GuestLAN,outside) source static GuestLAN GuestLAN destination static XXXXXXX-XXXX XXXXX-XXXX

nat (GuestLAN,outside) source static GuestLAN GuestLAN destination static XXXXXXX-XXXX XXXXX-XXXX

nat (GuestLAN,outside) source static GuestLAN GuestLAN destination static XXXXXXX-XXXX XXXXX-XXXX

nat (GuestLAN,outside) source static GuestLAN GuestLAN destination static XXXXXXX-XXXX XXXXX-XXXX

nat (GuestLAN,outside) source static GuestLAN GuestLAN destination static XXXXXXX-XXXX XXXXX-XXXX

nat (GuestLAN,outside) source static GuestLAN GuestLAN destination static XXXXXXX-XXXX XXXXX-XXXX

!

object network GuestLAN

nat (GuestLAN,outside) dynamic interface

FW#

FW# show nat

Manual NAT Policies (Section 1)

1 (inside) to (outside) source static OfficeLAN OfficeLAN   destination static XXXXXXX-XXXX XXXXX-XXXX

    translate_hits = 4586, untranslate_hits = 15

2 (inside) to (outside) source static OfficeLAN OfficeLAN   destination static XXXXXXX-XXXX XXXXX-XXXX

    translate_hits = 88, untranslate_hits = 0

3 (inside) to (outside) source static OfficeLAN OfficeLAN   destination static XXXXXXX-XXXX XXXXX-XXXX

    translate_hits = 344, untranslate_hits = 199

4 (inside) to (outside) source static OfficeLAN OfficeLAN   destination static XXXXXXX-XXXX XXXXX-XXXX

    translate_hits = 23007, untranslate_hits = 1197

5 (inside) to (outside) source dynamic any interface 

    translate_hits = 217724846, untranslate_hits = 4406

6 (inside) to (outside_backup) source dynamic any interface 

    translate_hits = 24616, untranslate_hits = 1877

7 (GuestLAN) to (outside) source static GuestLAN GuestLAN   destination static XXXXXXX-XXXX XXXXX-XXXX

    translate_hits = 751, untranslate_hits = 0

8 (GuestLAN) to (outside) source static GuestLAN GuestLAN   destination static XXXXXXX-XXXX XXXXX-XXXX

    translate_hits = 2708812, untranslate_hits = 1

9 (GuestLAN) to (outside) source static GuestLAN GuestLAN   destination static XXXXXXX-XXXX XXXXX-XXXX

    translate_hits = 87990, untranslate_hits = 0

10 (GuestLAN) to (outside) source static GuestLAN GuestLAN   destination static XXXXXXX-XXXX XXXXX-XXXX

    translate_hits = 88, untranslate_hits = 0

11 (GuestLAN) to (outside) source static GuestLAN GuestLAN   destination static XXXXXXX-XXXX XXXXX-XXXX

    translate_hits = 1539, untranslate_hits = 0

12 (GuestLAN) to (outside) source static GuestLAN GuestLAN   destination static XXXXXXX-XXXX XXXXX-XXXX

    translate_hits = 0, untranslate_hits = 0

13 (GuestLAN) to (outside) source static GuestLAN GuestLAN   destination static XXXXXXX-XXXX XXXXX-XXXX

    translate_hits = 0, untranslate_hits = 0

14 (GuestLAN) to (outside) source static GuestLAN GuestLAN   destination static XXXXXXX-XXXX XXXXX-XXXX

    translate_hits = 989, untranslate_hits = 0

Auto NAT Policies (Section 2)

1 (GuestLAN) to (outside) source dynamic GuestLAN interface 

    translate_hits = 5171112, untranslate_hits = 370

FW#   

Any help appreciated.

[Andrew Phirsov]

I think it might happen when traffic from remote subnet hits one of nat-exemtion rules (as it should) but returning traffic somehow hits dynamic pat rule, not the nat exemtion (if it's possible, though it shouldnt do so)

Try changing this two entries

nat (inside,outside) source dynamic any interface

nat (inside,outside_backup) source dynamic any interface

to this

nat (inside,outside) after-auto source dynamic any interface

nat (inside,outside_backup) after-auto source dynamic any interface

[Jouni Forss · Accepted solution]

Hi,

Only thing I have not done related to your environment would probably be the setup with 2 "outside" links. We handle 2 ISP setups with a single firewall interface while a routers or our core handles the dual ISP setup.

Which leads me to believe that there might be something wrong with your 2 "outside" interface operations perhaps?

The above configuration should probably handle the Internet traffic just fine but you have no corresponding NAT configurations for the L2L VPN NATs related to the other "outside" interface.

Maybe the setup is working for your Internet traffic (as you say) but when the main "outside" goes down there would be no NAT configurations to handle the L2L VPN traffic through the other "outside" link.

Also I think your software might already have an option to use "route-lookup" in the configuration to determine which eggress interface is used. To my understanding your current NAT0 / Identity NAT doesnt use route-looup. So If I am not mistaken even if you had NAT0 / Identity NAT for the other "outside" interface you might need "route-lookup" configured on both of them.

And theres ofcourse the L2L VPN configurations themselves.

But this is just guessing. As I mentioned I havent done 2 "outside" setups on the ASA itself since we handle them elsewhere. You could perhaps post some of the logs messages you are seeing?

- Jouni

[Jouni Forss]

Also,

As Andrew pointed out, it might not hurt to redo the NAT ordering of your firewall since at the moment the Default PAT rules might override some of your new NAT rules that you configure and you will have to keep an eye where to insert new rules. In general I wouldnt insert NAT rules are the "default" rules for some traffic (Internet in this case) to so high in the priorty of the ASAs NAT configurations.

When configured with the "after-auto" parameter they are inserted into Section 3 where they wont affect the more specific NAT rules.

- Jouni

[Andrey Padiy]

Thanks both.

Jouni you raised an interesting point regarding our second outside interface.

It's a standby ADSL line that is only used when primary goes down. Our ASA is configured with

route outside 0.0.0.0 0.0.0.0 X.X.X.X 1 track 1

route outside_backup 0.0.0.0 0.0.0.0 Y.Y.Y.Y 254

and

sla monitor 123

type echo protocol ipIcmpEcho X.X.X.X interface outside

sla monitor schedule 123 life forever start-time now

I am now wondering if the cause is outside route failover to outside_backup.

We have no L2L VPN configuration in place for outside_backup interface so NAT exemption rules for outside_backup are not needed.

Will take a closer look at the routes when it happens again.

[Jouni Forss]

Hi,

Heres one link regarding the changes which have been done in the latest 8 series software. And more specifically between its Maintanance releases 8.4(1) to 8.4(2)

http://packetpushers.net/understanding-when-a-cisco-asa-nat-rule-can-override-the-asa-routing-table/

This is more related to the NAT0 / Identity NAT operation in your software level and how that acts with the Egress interface decision and Route lookups.

- Jouni

[Jouni Forss]

Hi,

I have tried lab this setup at home for some time now and I cant really confirm what is stated in the above sites text.

If I understood the situation correctly then when configuring Identity NAT for example for L2L VPN then in software level 8.4(2) the ASA should by default use the NAT configuration to determine the eggress interface for some certain traffic.

In my case it seems that Route Lookup is done even if its not enable in the NAT configuration.

I configure an ASA5520 with 8.4(2) with Dual ISP and ASA 5505 8.2(1) as the L2L VPN Remote Site device.

• I first configured the Dual ISP setup on the ASA5520 and confirmed it was working ok for normal outbound traffic.
• I then configured the ASA5505 and confirmed its connectivity
• I then attached hosts behind both ASAs
• I then configured L2L VPN between the sites while configuring ASA5505 with 2 Peer IPs, both of the WAN IPs of the ASA 5520
• I then configured the Identity NAT from the ASA5520 LAN to the Remote Site LAN for both of the WAN interfaces of the ASA5520
• I remotely blocked the ICMP echo used to monitor the ISP1 link on ASA5520 with the result that 
  • ISP1 default route was switched to ISP2 default route
  • L2L VPN connection switched to use the ISP2 link
  • Traffic from the LAN behind ASA5520 started using the ISP2 link Identity NAT configuration istead of the ISP1 one that is configured before it

To my understanding the above behaviour seems to contradict the text in the above link I posted.

It states that starting from 8.4(2) NAT decides the eggress interface and NOT the Route Lookup. (Though you could configure "route-lookup" separately) In my case wether I configured "route-lookup" or not made no difference. Traffic was still switching between the NAT rules and more specifically the destination/eggress interfaces according to the routing table changes.

Could someone tell me if I am missing something here

According to the above test Andrey, you should be able to make sure that the L2L VPN would work through the other ISP link simply by doing the following things.

• Configure Identity NAT / NAT0 for the other "outside" interface
• Confirm that the remote site has L2L VPN configurations for both of your WAN IPs 
  • crypto map
  • tunnel-group configurations
  • tunnel-group configurations

EDIT:

I first thought the L2L VPN configuration had something to do with this operation and then I built an Identity NAT configuration which destination address isnt behind the L2L VPN and the results were the same. The NAT rule being used got switches as soon as the changes in the default route happend on the ASA.

- Jouni

Message was edited by: Jouni Forss

[Jouni Forss]

For the sake of testing I decided to boot the ASA5520 with 8.4(5) software.

After the boot the latest 8.4 software level it would seem that the NAT is NOW behaving the way the Cisco document states but still leaves the question why does the document state that this should already be the case in software 8.4(2)

The release notes of 8.4(2) already state this change in NAT operation

Identity NAT configurable proxy ARP and route lookup

In earlier releases for identity NAT, proxy ARP was disabled, and a  route lookup was always used to determine the egress interface. You  could not configure these settings. In 8.4(2) and later, the default  behavior for identity NAT was changed to match the behavior of other  static NAT configurations: proxy ARP is enabled, and the NAT  configuration determines the egress interface (if specified) by default.  You can leave these settings as is, or you can enable or disable them  discretely. Note that you can now also disable proxy ARP for regular  static NAT. For pre-8.3 configurations, the migration of NAT exempt rules (the nat 0 access-list command) to 8.4(2) and later now includes the following keywords to disable proxy ARP and to use a route lookup: no-proxy-arp and route-lookup. The unidirectional keyword that was used for migrating to 8.3(2) and 8.4(1) is no longer  used for migration. When upgrading to 8.4(2) from 8.3(1), 8.3(2), and  8.4(1), all identity NAT configurations will now include the no-proxy-arp and route-lookup keywords, to maintain existing functionality. The unidirectional keyword is removed. We modified the following commands: nat static [no-proxy-arp] [route-lookup] (object network) and nat source static [no-proxy-arp] [route-lookup] (global).

Source:

http://www.cisco.com/en/US/docs/security/asa/asa84/release/notes/asarn84.html#wp535067

So I kind of start to wonder if this is somehow related to some of the bugs in the new Twice NAT / Manual NAT configurations formats.

- Jouni

[Julio Carvajal]

Hello,

I just confirmed the same behavior than you Jouni, I ran a lab and got the same result,

Routing taking place...

[Andrey Padiy]

Very interesting. So if I understand this correctly you think we are seen this behaviour because the pimary route goes down?

In that case why do VPN tunnels stay up? When this happens all of our VPN tunnels are active with hours of session time on each. Surely if the default route fails over to outside_backup, IPSEC traffic would be routed out via outside_backup interface and peer IP on our end would be different which should cause IKE phase 1 to fail and L2L VPN session to tear down. Or am I missing something?

Many thanks.  

[Jouni Forss]

Hi,

At this point I can only speculate.

In many situation I think a L2L VPN can stay in UP state even though the actual connection isnt working. We dont know what exactly is happening in your situation since we have only seen the NAT configurations. Impossible to do anything else than guess. I dont know if the whole "outside" interface goes down or if there is something else that simply prevents the ICMP tracking from working.

I cant see how your VPN configurations are done. For example is the cryptomap attached to both of the "outside" interfaces and if "crypto ikev1 enable outside_backup" is configured. I can only guess that there is a possibility that if you dont have VPN configurations for the other "outside_backup" interface that the L2L VPN might actually hang on the "outside" interface even though the connection aint working.

As I have said before, I have never had the need to setup a Dual ISP with ASA before so I have never done this in production environment and therefore have never had to troubleshoot it and therefore cant give you an answer right here and now. I did lab this setup at home and still have it there.

What I can say according to the above NAT configuration is that you dont have the L2L VPN related NAT configurations for the "outside_backup" interface so because of that atleast the L2L VPN wont work.

- Jouni

[Andrey Padiy]

Looks like dual ISP setup was the culprit. I ve increased the tolerance level on the primary route monitoring and we havent had any issues since.

Many thanks for you help.