%ASA-5-305013: Asymmetric NAT rules matched for forward and reverse flows; Connection denied due to NAT reverse path failure. VPN client issues after 8.3 upgrade.

Answered Question
Sep 2nd, 2010

Hello,

We have ASA 5510 & upgraded to 8.3 recently. Now we are having

some issues accessing to our network. I have already read

https://supportforums.cisco.com/docs/DOC-12569 and

https://supportforums.cisco.com/message/3168125#3168125

but didn't help.

Our networks are

Trust (Inside): 172.28.10.0/24

DMZ: 172.28.5.0/24

VPN Pool: 172.28.8.0/24

From remote client, we can access to any hosts in inside segment,

however we cannot access any hosts in DMZ segment. when I ping

to a host in DMZ segment, I get following log.

Asymmetric NAT rules matched for forward and reverse flows; Connection for icmp

src Untrust:172.28.8.109 dst DMZ:172.28.5.10 (type 8, code 0) denied due to

NAT reverse path failure

here is the log of sh nat

===========================================================================

#sh nat
Manual NAT Policies (Section 1)
1 (Trust) to (any) source static DCT DCT destination static DLST DLST
     translate_hits = 360138, untranslate_hits = 78801
2 (Trust) to (any) source static DCT DCT destination static DLSV DLSV
     translate_hits = 116, untranslate_hits = 8992
3 (Trust) to (any) source static DCT DCT destination static DLSD DLSD
     translate_hits = 8, untranslate_hits = 0
4 (Trust) to (any) source static DDCT DCT destination static DETT DETT
     translate_hits = 1309, untranslate_hits = 52569
5 (Trust) to (any) source static DCT DCT destination static DETD DETD
     translate_hits = 35, untranslate_hits = 0
6 (Trust) to (any) source static DCT DCT destination static CPNT CPNT
     translate_hits = 118052, untranslate_hits = 44007
7 (DMZ) to (Untrust) source static DCD DCD destination static DLST DLST
     translate_hits = 9, untranslate_hits = 24
8 (DMZ) to (Untrust) source static DCD DCD destination static DLSV DLSV
     translate_hits = 0, untranslate_hits = 4
9 (DMZ) to (Untrust) source static DCD DCD destination static DLSD DLSD
     translate_hits = 17, untranslate_hits = 3
10 (DMZ) to (Untrust) source static DCD DCD destination static DETT DETT
     translate_hits = 2, untranslate_hits = 20
11 (DMZ) to (Untrust) source static DCD DCD destination static DETD DETD
     translate_hits = 2, untranslate_hits = 0
12 (DMZ) to (Untrust) source static DCD DCD destination static CPNT CPNT
     translate_hits = 22, untranslate_hits = 85
13 (Trust) to (Untrust) source static DCT DCT destination static our_company our_company
     translate_hits = 13, untranslate_hits = 2636
14 (Trust) to (Untrust) source static any any destination static DCV DCV
     translate_hits = 650, untranslate_hits = 11466
15 (DMZ) to (Untrust) source static any any destination static DCV DCV
     translate_hits = 0, untranslate_hits = 0

Auto NAT Policies (Section 2)
1 (DMZ) to (Untrust) source static FT01 "external IP 3" service tcp ftp ftp
     translate_hits = 0, untranslate_hits = 0
2 (DMZ) to (Untrust) source static Barracuda "external IP 2"
     translate_hits = 0, untranslate_hits = 31884
3 (Trust) to (Untrust) source static EX01 "external IP 1"
     translate_hits = 1245, untranslate_hits = 95428
4 (Trust) to (Untrust) source static MT01 "external IP 3" service tcp www www
     translate_hits = 0, untranslate_hits = 200
5 (Trust) to (Untrust) source dynamic obj_any interface
     translate_hits = 27787, untranslate_hits = 1325
6 (DMZ) to (Untrust) source dynamic obj_any-01 interface
     translate_hits = 719, untranslate_hits = 20

===========================================================================

Object:

DCT: 172.28.10.0/24

DCD: 172.28.5.0/24

DCV: 172.28.8.0/24

DLST: 172.28.40.0/24

DLSD: 172.28.63.0/24

DLSV: 172.28.62.0/24

DETT: 172.28.52.0/24

DETD: 172.28.64.0/24

CPNT: 172.28.30.0/24

FT01: 172.28.5.10

Barracuda: 172.28.5.250

EX01: 172.28.10.12

MT01: 172.28.10.14

I have this problem too.
0 votes
Correct Answer by Kureli Sankar about 6 years 2 months ago

This case is now resolved.

Due to this defect http://tools.cisco.com/Support/BugToolKit/search/getBugDetails.do?method=fetchBugDetails&bugId=CSCth24401

we could not find which nat rule was overlapping up above.

Symptom:
Packet tracer doesn't indicate all the overlapping NAT configurations when there is an rpf-check failure.

Conditions:
This is a basic functionality of packet tracers.

Workaround:
There is no known workaround.

We removed this line

nat (DMZ,Untrust) source static DC_DMZ DC_DMZ destination static DC_VPN DC_VPN

and added it as line 1

nat (DMZ,Untrust) 1 source static DC_DMZ DC_DMZ destination static DC_VPN DC_VPN

-KS

  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 5 (1 ratings)
Loading.
praprama Thu, 09/02/2010 - 18:39

Hi,

Can you attach the output of "show run nat". I am not sure why it's not working as the below rule should be taking effect:

15 (DMZ) to (Untrust) source static any any destination static DCV  DCV
     translate_hits = 0, untranslate_hits = 0

Regards,

Prapanch

Kureli Sankar Thu, 09/02/2010 - 18:48

Could you pls. collect the output of the below command

  packet-tracer DMZ inside icmp 172.28.5.10 0 4 172.28.8.109 det

-KS

ny_takeshi Fri, 09/03/2010 - 15:43

Hello kusankar,

Thank you very much for your kind reply.

Unfortunately the command didn't run.

I replaced IP, interface to

packet-tracer DMZ Trust icmp 172.28.5.10 0 4 172.28.8.113 det

but got error on DMZ, for some reason...

Thank you very much.

Kureli Sankar Fri, 09/03/2010 - 15:48

Sorry  I meant to say

packet-tracer DMZ icmp 172.28.5.10 0 4 172.28.8.113 det

-KS

Nagaraja Thanthry Fri, 09/03/2010 - 16:02

Hello,

A keyword "input" is missing:

packet-tracer input Trust icmp 172.28.5.10 0 4 172.28.8.113 det

Regards,

NT

ny_takeshi Fri, 09/03/2010 - 15:39

Thank you very much for your kind reply.

Here is the result of "sh run nat"

# sh run nat

nat (Trust,any) source static DCT DCT destination static DLST DLST

nat (Trust,any) source static DCT DCT destination static DLSV DLSV

nat (Trust,any) source static DCT DCT destination static DLSD DLSD

nat (Trust,any) source static DCT DCT destination static DETT DETT

nat (Trust,any) source static DCT DCT destination static DETD DETD

nat (Trust,any) source static DCT DCT destination static CPNT CPNT

nat (DMZ,Untrust) source static DCD DCD destination static DLST DLST

nat (DMZ,Untrust) source static DCD DCD destination static DLSV DLSV

nat (DMZ,Untrust) source static DCD DCD destination static DLSD DLSD

nat (DMZ,Untrust) source static DCD DCD destination static DETT DETT

nat (DMZ,Untrust) source static DCD DCD destination static DETD DETD

nat (DMZ,Untrust) source static DCD DCD destination static CPNT CPNT

nat (Trust,Untrust) source static DCT DCT destination static "our_company" "our_company"

nat (Trust,Untrust) source static any any destination static DCV DCV

nat (DMZ,Untrust) source static any any destination static DCV DCV

!

object network MT01

nat (Trust,Untrust) static public IP 3 service tcp www www

object network EX01

nat (Trust,Untrust) static public IP 1

object network obj_any

nat (Trust,Untrust) dynamic interface

object network obj_any-01

nat (DMZ,Untrust) dynamic interface

object network FT01

nat (DMZ,Untrust) static public IP 3 service tcp ftp ftp

object network Barracuda

nat (DMZ,Untrust) static Public IP 2

Thank you very much

ny_takeshi Fri, 09/03/2010 - 16:04

Hello kusankar

it didn't work, but i worked it out.

here is the result.

# packet-tracer input Trust icmp 172.28.5.10 0 4 172.28.8.113 det

Phase: 1

Type: ROUTE-LOOKUP

Subtype: input

Result: ALLOW

Config:

Additional Information:

in   172.28.8.113    255.255.255.255 Untrust

Phase: 2

Type: IP-OPTIONS

Subtype:

Result: ALLOW

Config:

Additional Information:

Forward Flow based lookup yields rule:

in  id=0xacba90f0, priority=0, domain=inspect-ip-options, deny=true

        hits=902666, user_data=0x0, cs_id=0x0, reverse, flags=0x0, protocol=0

        src ip/id=0.0.0.0, mask=0.0.0.0, port=0

        dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, dscp=0x0

        input_ifc=Trust, output_ifc=any

Phase: 3

Type: INSPECT

Subtype: np-inspect

Result: ALLOW

Config:

class-map inspection_default

match default-inspection-traffic

policy-map global_policy

class inspection_default

  inspect icmp

service-policy global_policy global

Additional Information:

Forward Flow based lookup yields rule:

in  id=0xad4044c0, priority=70, domain=inspect-icmp, deny=false

        hits=356641, user_data=0xad404818, cs_id=0x0, use_real_addr, flags=0x0, protocol=1

        src ip/id=0.0.0.0, mask=0.0.0.0, port=0

        dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, dscp=0x0

        input_ifc=Trust, output_ifc=any

Phase: 4

Type: INSPECT

Subtype: np-inspect

Result: ALLOW

Config:

Additional Information:

Forward Flow based lookup yields rule:

in  id=0xacba8d58, priority=66, domain=inspect-icmp-error, deny=false

        hits=384117, user_data=0xacba8c40, cs_id=0x0, use_real_addr, flags=0x0, protocol=1

        src ip/id=0.0.0.0, mask=0.0.0.0, port=0

        dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, dscp=0x0

        input_ifc=Trust, output_ifc=any

Phase: 5

Type: NAT

Subtype:

Result: ALLOW

Config:

nat (Trust,Untrust) source static any any destination static DC_VPN DC_VPN

Additional Information:

Static translate MEC-FT01/0 to MEC-FT01/0

Forward Flow based lookup yields rule:

in  id=0xad6470f0, priority=6, domain=nat, deny=false

        hits=2255, user_data=0xad646a80, cs_id=0x0, use_real_addr, flags=0x0, protocol=0

        src ip/id=0.0.0.0, mask=0.0.0.0, port=0

        dst ip/id=172.28.8.0, mask=255.255.255.0, port=0, dscp=0x0

        input_ifc=Trust, output_ifc=Untrust

Phase: 6

Type: VPN

Subtype: encrypt

Result: ALLOW

Config:

Additional Information:

Forward Flow based lookup yields rule:

out id=0xad593b50, priority=70, domain=encrypt, deny=false

        hits=1, user_data=0x855e454, cs_id=0x0, reverse, flags=0x0, protocol=0

        src ip/id=0.0.0.0, mask=0.0.0.0, port=0

        dst ip/id=172.28.8.113, mask=255.255.255.255, port=0, dscp=0x0

        input_ifc=any, output_ifc=Untrust

Phase: 7

Type: FLOW-CREATION

Subtype:

Result: ALLOW

Config:

Additional Information:

New flow created with id 1006626, packet dispatched to next module

Module information for forward flow ...

snp_fp_tracer_drop

snp_fp_inspect_ip_options

snp_fp_inspect_icmp

snp_fp_translate

snp_fp_adjacency

snp_fp_encrypt

snp_fp_fragment

snp_ifc_stat

Module information for reverse flow ...

Result:

input-interface: Trust

input-status: up

input-line-status: up

output-interface: Untrust

output-status: up

output-line-status: up

Action: allow

Kureli Sankar Fri, 09/03/2010 - 16:40

Trust : 172.28.10.0/24

DMZ: 172.28.5.0/24

VPN Pool: 172.28.8.0/24

Asymmetric NAT rules matched for forward and reverse flows; Connection for icmp src Untrust:172.28.8.109 dst DMZ:172.28.5.10 (type 8, code 0) denied due to NAT reverse path failure

packet-tracer input DMZ icmp 172.28.5.10 0 4 172.28.8.113 det

packet-tracer input Untrust icmp 172.28.8.113 0 8  172.28.5.10 det

One more time. The above two outputs pls. Just "?" mark your way if the commands don't take as we type it on the fly and I apologize for missing the "input" keyword that NT caught very quickly.

BTW,

I don't see this line nat (Trust,Untrust) source static any any destination static DC_VPN DC_VPN

in the sh run nat output how come?

-KS

ny_takeshi Tue, 09/07/2010 - 08:49

Hello Kusankar,

here are the log for the commands.

====================================================================================

# packet-tracer input DMZ icmp 172.28.5.10 0 4 172.28.8.113 det

Phase: 1

Type: ROUTE-LOOKUP

Subtype: input

Result: ALLOW

Config:

Additional Information:

in   172.28.8.113    255.255.255.255 Untrust

Phase: 2

Type: IP-OPTIONS

Subtype:

Result: ALLOW

Config:

Additional Information:

Forward Flow based lookup yields rule:

in  id=0xacbcf428, priority=0, domain=inspect-ip-options, deny=true

        hits=64585, user_data=0x0, cs_id=0x0, reverse, flags=0x0, protocol=0

        src ip/id=0.0.0.0, mask=0.0.0.0, port=0

        dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, dscp=0x0

        input_ifc=DMZ, output_ifc=any

Phase: 3

Type: INSPECT

Subtype: np-inspect

Result: ALLOW

Config:

class-map inspection_default

match default-inspection-traffic

policy-map global_policy

class inspection_default

  inspect icmp

service-policy global_policy global

Additional Information:

Forward Flow based lookup yields rule:

in  id=0xad405e58, priority=70, domain=inspect-icmp, deny=false

        hits=134, user_data=0xad404818, cs_id=0x0, use_real_addr, flags=0x0, pro

tocol=1

        src ip/id=0.0.0.0, mask=0.0.0.0, port=0

        dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, dscp=0x0

        input_ifc=DMZ, output_ifc=any

Phase: 4

Type: INSPECT

Subtype: np-inspect

Result: ALLOW

Config:

Additional Information:

Forward Flow based lookup yields rule:

in  id=0xacbcf090, priority=66, domain=inspect-icmp-error, deny=false

        hits=134, user_data=0xacbcef78, cs_id=0x0, use_real_addr, flags=0x0, pro

tocol=1

        src ip/id=0.0.0.0, mask=0.0.0.0, port=0

        dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, dscp=0x0

        input_ifc=DMZ, output_ifc=any

Phase: 5

Type: VPN

Subtype: ipsec-tunnel-flow

Result: ALLOW

Config:

Additional Information:

Forward Flow based lookup yields rule:

in  id=0xad631098, priority=13, domain=ipsec-tunnel-flow, deny=true

        hits=83, user_data=0x0, cs_id=0x0, flags=0x0, protocol=0

        src ip/id=0.0.0.0, mask=0.0.0.0, port=0

        dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, dscp=0x0

        input_ifc=DMZ, output_ifc=any

Phase: 6

Type: NAT

Subtype:

Result: ALLOW

Config:

nat (DMZ,Untrust) source static any any destination static DCV DCV

Additional Information:

Static translate MEC-FT01/0 to MEC-FT01/0

Forward Flow based lookup yields rule:

in  id=0xad499c20, priority=6, domain=nat, deny=false

        hits=1, user_data=0xad582f20, cs_id=0x0, use_real_addr, flags=0x0, protocol=0

        src ip/id=0.0.0.0, mask=0.0.0.0, port=0

        dst ip/id=172.28.8.0, mask=255.255.255.0, port=0, dscp=0x0

        input_ifc=DMZ, output_ifc=Untrust

Phase: 7

Type: VPN

Subtype: encrypt

Result: ALLOW

Config:

Additional Information:

Forward Flow based lookup yields rule:

out id=0xac87ded0, priority=70, domain=encrypt, deny=false

        hits=1, user_data=0x99127ec, cs_id=0x0, reverse, flags=0x0, protocol=0

        src ip/id=0.0.0.0, mask=0.0.0.0, port=0

        dst ip/id=172.28.8.113, mask=255.255.255.255, port=0, dscp=0x0

        input_ifc=any, output_ifc=Untrust

Phase: 8

Type: FLOW-CREATION

Subtype:

Result: ALLOW

Config:

Additional Information:

New flow created with id 1150310, packet dispatched to next module

Module information for forward flow ...

snp_fp_tracer_drop

snp_fp_inspect_ip_options

snp_fp_inspect_icmp

snp_fp_translate

snp_fp_adjacency

snp_fp_encrypt

snp_fp_fragment

snp_ifc_stat

Module information for reverse flow ...

Result:

input-interface: DMZ

input-status: up

input-line-status: up

output-interface: Untrust

output-status: up

output-line-status: up

Action: allow

====================================================================================

# packet-tracer input Untrust icmp 172.28.8.113 0 8  172.28.5.10 det

Phase: 1

Type: ACCESS-LIST

Subtype:

Result: ALLOW

Config:

Implicit Rule

Additional Information:

Forward Flow based lookup yields rule:

in  id=0xacad83e0, priority=1, domain=permit, deny=false

        hits=34963188, user_data=0x0, cs_id=0x0, l3_type=0x8

        src mac=0000.0000.0000, mask=0000.0000.0000

        dst mac=0000.0000.0000, mask=0100.0000.0000

        input_ifc=Untrust, output_ifc=any

Phase: 2

Type: ROUTE-LOOKUP

Subtype: input

Result: ALLOW

Config:

Additional Information:

in   172.28.8.113    255.255.255.255 Untrust

Phase: 3

Type: ACCESS-LIST

Subtype:

Result: DROP

Config:

Implicit Rule

Additional Information:

Forward Flow based lookup yields rule:

in  id=0xa7f80050, priority=11, domain=permit, deny=true

        hits=182336, user_data=0x5, cs_id=0x0, flags=0x0, protocol=0

        src ip/id=0.0.0.0, mask=0.0.0.0, port=0

        dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, dscp=0x0

        input_ifc=Untrust, output_ifc=any

Result:

input-interface: Untrust

input-status: up

input-line-status: up

output-interface: Untrust

output-status: up

output-line-status: up

Action: drop

Drop-reason: (acl-drop) Flow is denied by configured rule

Thank you very much!
Regards,
Tak
ny_takeshi Wed, 09/08/2010 - 12:46

Hello Kusankar,

I forgot to reply to your question.

>I don't see this line nat (Trust,Untrust) source static any any destination static DC_VPN DC_VPN

I modified the original config a bit. just object names. DC_VPN = DCV

Thanks,


Takeshi

Kureli Sankar Sat, 09/11/2010 - 18:10

Takeshi,

Sorry for the delay.

packet-tracer input DMZ icmp 172.28.5.10 0 4 172.28.8.113 det
Result:
input-interface: DMZ
output-interface: Untrust
Action: allow
nat (DMZ,Untrust) source static any any destination static DCV DCV

The above output looks ok.


packet-tracer input Untrust icmp 172.28.8.113 0 8  172.28.5.10

Result:
input-interface: Untrust
output-interface: Untrust
Action: drop

The above output says the source and the destination are out the Untrust interface.

Pls. check the route. What do your "sh route | i 172.28.5.0" and "sh route | i 172.28.8.0" say? Do they show the correct interface?

-KS

ny_takeshi Tue, 09/14/2010 - 16:19

Hello Kusankar,

Thank you very much for your kind attention & comments.

I ran the command you specified below, but didn't return anything.

just in case, I just ran "sh route" without output modifier.

==========================================================================

# sh route

Codes: C - connected, S - static, I - IGRP, R - RIP, M - mobile, B - BGP

       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area

       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2

       E1 - OSPF external type 1, E2 - OSPF external type 2, E - EGP

       i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area

       * - candidate default, U - per-user static route, o - ODR

       P - periodic downloaded static route

Gateway of last resort is "GW IP" to network 0.0.0.0

S    172.17.0.0 255.255.0.0 [1/0] via 172.28.10.1, Trust

S    172.16.0.0 255.255.0.0 [1/0] via 172.28.10.1, Trust

S    172.19.0.0 255.255.0.0 [1/0] via 172.28.10.1, Trust

S    172.18.0.0 255.255.0.0 [1/0] via 172.28.10.1, Trust

S    172.21.0.0 255.255.0.0 [1/0] via 172.28.10.1, Trust

S    172.20.0.0 255.255.0.0 [1/0] via 172.28.10.1, Trust

S    172.24.0.0 255.255.0.0 [1/0] via 172.28.10.1, Trust

S    172.27.0.0 255.255.0.0 [1/0] via 172.28.10.1, Trust

C    DCD 255.255.255.0 is directly connected, DMZ

C    DCT 255.255.255.0 is directly connected, Trust

S    172.28.8.116 255.255.255.255 [1/0] via "GW IP", Untrust  --> Currently someone is accessing via VPN

S    172.28.8.118 255.255.255.255 [1/0] via "GW IP", Untrust  --> Currently someone is accessing via VPN

S    172.28.8.108 255.255.255.255 [1/0] via "GW IP", Untrust  --> Currently someone is accessing via VPN

C    "Public Network Address" 255.255.255.248 is directly connected, Untrust

S*   0.0.0.0 0.0.0.0 [1/0] via "GW IP", Untrust

==========================================================================

Thank you very much in advance!!
Regards,
Takeshi

Kureli Sankar Mon, 09/20/2010 - 20:33

Takeshi,

I suggest you open a TAC case as we need to collect a bunch of output while ping -t is going on between the two hosts.

The route looks ok but, still the packet tracker output is clearly getting dropped.

-KS

ny_takeshi Tue, 09/21/2010 - 08:59

Hello Kusankar,

Thank you very much for your kind assistance.

I have created a TAC case. Hope they can help me.

Again, Thank you sooo much for your help.

Best Regards,

Takeshi

Correct Answer
Kureli Sankar Tue, 09/21/2010 - 11:53

This case is now resolved.

Due to this defect http://tools.cisco.com/Support/BugToolKit/search/getBugDetails.do?method=fetchBugDetails&bugId=CSCth24401

we could not find which nat rule was overlapping up above.

Symptom:
Packet tracer doesn't indicate all the overlapping NAT configurations when there is an rpf-check failure.

Conditions:
This is a basic functionality of packet tracers.

Workaround:
There is no known workaround.

We removed this line

nat (DMZ,Untrust) source static DC_DMZ DC_DMZ destination static DC_VPN DC_VPN

and added it as line 1

nat (DMZ,Untrust) 1 source static DC_DMZ DC_DMZ destination static DC_VPN DC_VPN

-KS

ny_takeshi Tue, 09/21/2010 - 12:06

Hello Kusankar,

Thank you very much for your great help!

Best Regards,

Takeshi

Actions

This Discussion