Inbound NAT issue with PBR on ASA

[tato386]

I have an ASA (9.6.3) with two interfaces connected to the Internet.  The ASA default route is pointing to ISP A and I have PAT and NAT using ISP A working fine.  I have a route-map using PBR that sets default next hop for certain clients to ISP B.  For the clients using ISP B I also have PAT and NAT setup.  PAT works fine and NAT works fine for _outbound_ traffic but I cannot get any inbound services to work. 

Test show that it is not a problem with rules or NAT because if I add a static route on the ASA that uses ISP B for a particular Internet IP the inbound works.  So I guess I need to add something else for NAT/PBR to work but I am not sure what.  Any ideas?

Thanks
Diego

[Flavio Miranda]

Hello @tato386

 Really looks like routing problem, probably asymmetric routing. Probably a capture will give you the answer.

 If possible, share you config here so that we can take a look.

-If I helped you somehow, please, rate it as useful.-

[Spooster IT Services]

Hi Diego,

Can you please send me the configuration related to PBR that you have done on ASA?

[tato386]

sanitized config:

ASA Version 9.6(3)1
!
interface GigabitEthernet0/0
 nameif inf_Data
 security-level 100
 ip address 10.1.1.254 255.255.255.0
 policy-route route-map ALT-GATEWAY
!
interface GigabitEthernet0/1
 desc /30 with /29 routeable block
 nameif inf_ISPB
 security-level 0
 ip address 2.2.2.2 255.255.255.252
!
interface GigabitEthernet0/5
 nameif inf_ISPA
 security-level 0
 ip address 1.1.1.2 255.255.255.248
!
!
object network host1
 host 10.1.1.20
object network net_ISPB-PublicBlock
 subnet 3.3.3.0 255.255.255.248
object network ip_ISPB-NAT
 host 3.3.3.1

access-list acl_Firewall-ISPA extended permit icmp any any

!

access-list acl_Firewall-ISPB extended permit icmp any any
access-list acl_Firewall-ISPB extended permit tcp any object host1 eq telnet

!

access-list acl_ISPB-PBR extended permit ip object host1 any4
access-list acl_ISPB-PBR extended deny ip any4 any4
!
!
object network host1
 nat (inf_Data,any) static ip_ISPB-NAT
!
access-group acl_Firewall-ISPB in interface inf_ISPB
access-group acl_Firewall-ISPA in interface inf_ISPA
!
route-map ALT-GATEWAY permit 10
 match ip address acl_ISPB-PBR
 set ip default next-hop 2.2.2.1
!
route inf_ISPA 0.0.0.0 0.0.0.0 1.1.1.1 1

[Spooster IT Services]

Hi Diego,

Can you please run packet tracer as mentioned below and share the output with us?

packet tracer input int_ISPB tcp 8.8.8.8 12121 3.3.3.1 23 detailed

[tato386]

The packet trace looks as it should.  The problem is that the ASA is trying to reply out of the wrong interface.  If I add a static route to 8.8.8.8 using inf_ISPB it works.  So it seems that PBR is respected when the inside host initiates a flow to the outside but it is not used for packets initiated from outside to inside hosts.  

asa#packet input inf_ISPB tcp 8.8.8.8 1212 3.3.3.1 23 detailed

Phase: 1
Type: UN-NAT
Subtype: static
Result: ALLOW
Config:
object network host1
 nat (inf_Data,any) static ip_Test
Additional Information:
NAT divert to egress interface inf_Data
Untranslate 3.3.3.1/23 to 10.1.1.20/23

Phase: 2
Type: ACCESS-LIST
Subtype: log
Result: ALLOW
Config:
access-group inf_ISPB_access_in in interface inf_ISPB
access-list inf_ISPB_access_in extended permit tcp any object host1 eq telnet
Additional Information:
 Forward Flow based lookup yields rule:
 in  id=0x2aaac3165830, priority=13, domain=permit, deny=false
        hits=948, user_data=0x2aaab97918c0, cs_id=0x0, use_real_addr, flags=0x0, protocol=6
        src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
        dst ip/id=10.1.1.20, mask=255.255.255.255, port=23, tag=any, dscp=0x0
        input_ifc=inf_ISPB, output_ifc=any

Phase: 3
Type: CONN-SETTINGS
Subtype:
Result: ALLOW
Config:
class-map class-default
 match any
policy-map global_policy
 class class-default
  set connection conn-max 0 embryonic-conn-max 0 random-sequence-number disable
service-policy global_policy global
Additional Information:
 Forward Flow based lookup yields rule:
 in  id=0x2aaac3076560, priority=7, domain=conn-set, deny=false
        hits=3658, user_data=0x2aaac3073670, cs_id=0x0, use_real_addr, flags=0x0, protocol=0
        src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
        dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
        input_ifc=inf_ISPB, output_ifc=any

Phase: 4
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:
 Forward Flow based lookup yields rule:
 in  id=0x2aaac099fcb0, priority=0, domain=nat-per-session, deny=false
        hits=1074533, user_data=0x0, cs_id=0x0, reverse, use_real_addr, flags=0x0, protocol=6
        src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
        dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
        input_ifc=any, output_ifc=any

Phase: 5
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:
 Forward Flow based lookup yields rule:
 in  id=0x2aaac137e530, priority=0, domain=inspect-ip-options, deny=true
        hits=3977, user_data=0x0, cs_id=0x0, reverse, flags=0x0, protocol=0
        src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
        dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
        input_ifc=inf_ISPB, output_ifc=any

Phase: 6
Type: NAT
Subtype: rpf-check
Result: ALLOW
Config:
object network host1
 nat (inf_Data,any) static ip_Test
Additional Information:
 Forward Flow based lookup yields rule:
 out id=0x2aaac40c3f00, priority=6, domain=nat-reverse, deny=false
        hits=972, user_data=0x2aaac40c5180, cs_id=0x0, use_real_addr, flags=0x0, protocol=0
        src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
        dst ip/id=10.1.1.20, mask=255.255.255.255, port=0, tag=any, dscp=0x0
        input_ifc=any, output_ifc=inf_Data

Phase: 7
Type: USER-STATISTICS
Subtype: user-statistics
Result: ALLOW
Config:
Additional Information:
 Forward Flow based lookup yields rule:
 out id=0x2aaac306c690, priority=0, domain=user-statistics, deny=false
        hits=1023068, user_data=0x2aaac2ffd2c0, cs_id=0x0, reverse, flags=0x0, protocol=0
        src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
        dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
        input_ifc=any, output_ifc=inf_Data

Phase: 8
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:
 Reverse Flow based lookup yields rule:
 in  id=0x2aaac099fcb0, priority=0, domain=nat-per-session, deny=false
        hits=1074535, user_data=0x0, cs_id=0x0, reverse, use_real_addr, flags=0x0, protocol=6
        src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
        dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
        input_ifc=any, output_ifc=any

Phase: 9
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:
 Reverse Flow based lookup yields rule:
 in  id=0x2aaac1317820, priority=0, domain=inspect-ip-options, deny=true
        hits=789889, user_data=0x0, cs_id=0x0, reverse, flags=0x0, protocol=0
        src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
        dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
        input_ifc=inf_Data, output_ifc=any

Phase: 10
Type: USER-STATISTICS
Subtype: user-statistics
Result: ALLOW
Config:
Additional Information:
 Reverse Flow based lookup yields rule:
 out id=0x2aaac306d630, priority=0, domain=user-statistics, deny=false
        hits=3259, user_data=0x2aaac2ffd2c0, cs_id=0x0, reverse, flags=0x0, protocol=0
        src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
        dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
        input_ifc=any, output_ifc=inf_ISPB

Phase: 11
Type: FLOW-CREATION
Subtype:
Result: ALLOW
Config:
Additional Information:
New flow created with id 1019957, packet dispatched to next module
Module information for forward flow ...
snp_fp_tracer_drop
snp_fp_inspect_ip_options
snp_fp_tcp_normalizer
snp_fp_translate
snp_fp_adjacency
snp_fp_fragment
snp_ifc_stat

Module information for reverse flow ...
snp_fp_tracer_drop
snp_fp_inspect_ip_options
snp_fp_translate
snp_fp_tcp_normalizer
snp_fp_adjacency
snp_fp_fragment
snp_ifc_stat

Result:
input-interface: inf_ISPB
input-status: up
input-line-status: up
output-interface: inf_Data
output-status: up
output-line-status: up
Action: allow

[Spooster IT Services]

Hi Diego,

Can you please make the following changes on the route map and test it?

route-map ALT-GATEWAY permit 10
match ip address acl_ISPB-PBR
no set ip default next-hop 2.2.2.1
set ip next-hop 2.2.2.1

If this still not working, then please take the captures of the traffic to find out the issue.
access-list test extended permit tcp any4 host 10.1.1.20 23
access-list test extended permit tcp host 10.1.1.20 23 any4
!
capture capi interface inf_Data access-list test
!

[tato386]

I adjusted the route-map as you suggested and it didn't make a difference.  I also played around with moving the NAT to "before object NAT" and that didn't make a difference.  I have attached the packet capture and it seems OK.  It doesn't show the translated public IP but I am sure that it working because I have tested it using sites like ipchicken.com.

I appreciate your help very much but I am starting to think this is a bug.  

[tato386]

According to TAC this is something that has worked in older versions but no longer available in newer ASA versions.  I am pretty sure I have done this in the past so it does not sound totally off base.  Not the answer I wanted to hear and very disappointing to have a useful feature removed.

Thanks to all who tried to help.

Diego

[dbogdan]

Did you ever get this to work?  I face the same issue when attempting to use a route-map.  I have to add the route for the route-map to receive traffic from the outside, which kinda defeats the purpose.  May as well just define a pile of routes instead.

Any advice would be appreciated!!

[tato386]

Sorry I was never able to get this to work but there have been several software updates to ASA since I was messing around with this. Have you tried using a recent build?  Maybe they changed the behavior back?

[chris-goulder · Accepted solution]

The 'old way' of making this type of setup work was to include a floating static route for the second internet path

Referring to your config above include: -

route inf_ISPB 0.0.0.0 0.0.0.0 2.2.2.2 100

This adds internet route to the table, that not used for normal traffic due to the higher metric but completes the picture for PBR / NAT inbound traffic flows

[tato386]

At this time I don't have a setup where I can test this but I surely appreciate the info.  It might come in handy at some point.

Thank you!

[Nishanna Gunasekera]

I can verify that this works. Thank you Chris!

[dbogdan]

Thanks for sorting this out. I will test at some future point.