VPN Clients connecting to a Spoke Network being blocked.

[jstevensunico]

Hi All,

We have a Site to Site VPN set up with a customer's network and all works well from our Workgroup LAN on the inside interface of an ASA (8.0(4).  The tunnel is created and traffic passes over it fine.  We also need remote users using the Anyconnect client to be able to access the customer's network.  what we see is that the tunnel is created, but traffic is denied.

This is traffic from the outside interface going back out of the outside interface.  The config parameter "same-security-traffic permit intra-interface" is in place.

A packet-trace of a ping shows:

5qr-5-fw2# packet-tracer input outside icmp 10.100.100.128 0 0 172.17.120.59 d$

Phase: 1
Type: FLOW-LOOKUP
Subtype:
Result: ALLOW
Config:
Additional Information:
Found no matching flow, creating a new flow

Phase: 2
Type: ROUTE-LOOKUP
Subtype: input
Result: ALLOW
Config:
Additional Information:
in   0.0.0.0         0.0.0.0         outside

Phase: 3
Type: ACCESS-LIST
Subtype: log
Result: ALLOW
Config:
access-group outside_access_in in interface outside
access-list outside_access_in extended permit icmp any any
Additional Information:
Forward Flow based lookup yields rule:
in  id=0xccbc74a8, priority=12, domain=permit, deny=false
        hits=824, user_data=0xcce7b300, cs_id=0x0, flags=0x0, protocol=1
        src ip=0.0.0.0, mask=0.0.0.0, port=0
        dst ip=0.0.0.0, mask=0.0.0.0, port=0, dscp=0x0

Phase: 4
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in  id=0xc89aeb58, priority=0, domain=permit-ip-option, deny=true
        hits=254576, user_data=0x0, cs_id=0x0, reverse, flags=0x0, protocol=0
        src ip=0.0.0.0, mask=0.0.0.0, port=0
        dst ip=0.0.0.0, mask=0.0.0.0, port=0, dscp=0x0

Phase: 5
Type: INSPECT
Subtype: np-inspect
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in  id=0xc89ae048, priority=66, domain=inspect-icmp-error, deny=false
        hits=3211, user_data=0xc89adf78, cs_id=0x0, use_real_addr, flags=0x0, protocol=1
        src ip=0.0.0.0, mask=0.0.0.0, port=0
        dst ip=0.0.0.0, mask=0.0.0.0, port=0, dscp=0x0

Phase: 6
Type: VPN
Subtype: ipsec-tunnel-flow
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in  id=0xc90bea28, priority=12, domain=ipsec-tunnel-flow, deny=true
        hits=86530, user_data=0x0, cs_id=0x0, flags=0x0, protocol=0
        src ip=0.0.0.0, mask=0.0.0.0, port=0
        dst ip=0.0.0.0, mask=0.0.0.0, port=0, dscp=0x0

Phase: 7
Type: NAT-EXEMPT
Subtype:
Result: ALLOW
Config:
nat-control
  match ip outside 10.100.100.128 255.255.255.128 outside Unico-TU-Desktops 255.255.255.0
    NAT exempt
    translate_hits = 423, untranslate_hits = 0
Additional Information:
Forward Flow based lookup yields rule:
in  id=0xcca8e158, priority=6, domain=nat-exempt, deny=false
        hits=422, user_data=0xccd07e08, cs_id=0x0, use_real_addr, flags=0x0, protocol=0
        src ip=10.100.100.128, mask=255.255.255.128, port=0
        dst ip=Unico-TU-Desktops, mask=255.255.255.0, port=0, dscp=0x0

Phase: 8
Type: NAT
Subtype:
Result: ALLOW
Config:
nat (outside) 1 10.100.100.0 255.255.255.0
nat-control
  match ip outside 10.100.100.0 255.255.255.0 outside any
    dynamic translation to pool 1 (165.228.101.70 [Interface PAT])
    translate_hits = 3337, untranslate_hits = 182
Additional Information:
Forward Flow based lookup yields rule:
in  id=0xc8a777d8, priority=1, domain=nat, deny=false
        hits=3803, user_data=0xc8a77738, cs_id=0x0, flags=0x0, protocol=0
        src ip=10.100.100.0, mask=255.255.255.0, port=0
        dst ip=0.0.0.0, mask=0.0.0.0, port=0, dscp=0x0

Phase: 9
Type: NAT
Subtype: host-limits
Result: ALLOW
Config:
nat (outside) 1 10.100.100.0 255.255.255.0
nat-control
  match ip outside 10.100.100.0 255.255.255.0 outside any
    dynamic translation to pool 1 (165.228.101.70 [Interface PAT])
    translate_hits = 3338, untranslate_hits = 182
Additional Information:
Forward Flow based lookup yields rule:
in  id=0xc8a77af0, priority=1, domain=host, deny=false
        hits=11897, user_data=0xc8a77738, cs_id=0x0, reverse, flags=0x0, protocol=0
        src ip=10.100.100.0, mask=255.255.255.0, port=0
        dst ip=0.0.0.0, mask=0.0.0.0, port=0, dscp=0x0

Phase: 10
Type: VPN
Subtype: encrypt
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
out id=0xccffa068, priority=70, domain=encrypt, deny=false
        hits=1, user_data=0x2ad9c, cs_id=0xc9044c78, reverse, flags=0x0, protocol=0
        src ip=10.100.100.0, mask=255.255.255.0, port=0
        dst ip=Unico-TU-Desktops, mask=255.255.255.0, port=0, dscp=0x0

Phase: 11
Type: ACCESS-LIST
Subtype: ipsec-user
Result: DROP
Config:
Additional Information:
Forward Flow based lookup yields rule:
out id=0xcccdc4a0, priority=69, domain=ipsec-user, deny=true
        hits=1, user_data=0x0, cs_id=0x0, flags=0x0, protocol=0
        src ip=10.100.100.0, mask=255.255.255.0, port=0
        dst ip=Unico-TU-Desktops, mask=255.255.255.0, port=0, dscp=0x0

Result:
input-interface: outside
input-status: up
input-line-status: up
output-interface: outside
output-status: up
output-line-status: up
Action: drop
Drop-reason: (acl-drop) Flow is denied by configured rule

All is find until Phase 11 when some Access-List drops it.  But how do I identify what access list is doing this?  It is most annoying.  I have tried adding ACEs to the outside interface to allow the traffic in both directions, but it just doesn't work.

Any assistance or thoughts are greatly appreciated.

Cheers

[busterswt]

You ought to be able to enable logging to catch which ACL is denying that traffic. Try the following:

logging enable

logging message 106100 level 6

logging buffer-size 100000

Generate some traffic and issue a 'sh log' to check the log for ACL hits.

If that logging message doesn't work, try issuing 'logging buffered 6' to get all level 6 logs and just parse through them. Don't forget to issue a 'no logging enable' when done.

Good luck,

James

[jstevensunico]

Hi James,

Thanks for the thoughts, but the logs do not show which ACL is being hit.

If the packet-trace output is to be belived, the ACL id is 0xcccdc4a0, which does not appear in a sh access-list output.  This would indicate that it is an implicit rule.  Problem is I cannot locate which interface it is is on or which ACL name it belongs to.  I would have thought that it was the outside_access_in rule, but adding ACEs to this ACL to allow the traffic does not work, and the ACEs show now hits.

I am at a complete loss.

Cheers

[yamramos.tueme]

Hi John!

Do you have any vpn-filter configured?  The packet tracke outuput suggest that is an ACL, but not the one applied to the interface

Type: ACCESS-LIST
Subtype: ipsec-user

Do you have any relevante configuration you can share?

- Yamil

[williamwarren17]

I know this is old by now, but I just spent days figuring this out. Try taking out your vpn-filter ACL like so:

no access-list vpn-filter permit/deny ip

ALSO, make sure your crypto ACL is set like this:

access-list OUTSIDE_CRYPTO extended permit ip host

Your source is your REMOTE addresses, and destination is your LOCAL. I scoured the internet and NO ONE had an answer for this. Seems like more people would have run into this at some point. Anyway, let me know!