Re: Problem establishing SSL VPN from only 1 IP address

Jernej Vodopivec · ‎11-25-2013

Hi,

I'm experiencing strange problem.

I can't establish SSL VPN connection from 1 IP address, but I don't have problem establishing SSL VPN from any other IP address.

Remote IP address: 10.0.0.1

ASA's public IP address: 192.168.1.1

Output of packet-tracer:

1. with problematic source IP address:

packet-tracer input wan tcp 10.0.0.1 50601 192.168.1.1 443 detailed

Phase: 1

Type: ROUTE-LOOKUP

Subtype: input

Result: ALLOW

Config:

Additional Information:

in 192.168.1.1 255.255.255.255 identity

Phase: 2

Type: ACCESS-LIST

Subtype:

Result: ALLOW

Config:

Implicit Rule

Additional Information:

Forward Flow based lookup yields rule:

in id=0x7fff37573f00, priority=119, domain=permit, deny=false

hits=861, user_data=0x0, cs_id=0x0, flags=0x0, protocol=6

src ip/id=0.0.0.0, mask=0.0.0.0, port=0

dst ip/id=0.0.0.0, mask=0.0.0.0, port=443, dscp=0x0

input_ifc=wan, output_ifc=identity

Phase: 3

Type: CONN-SETTINGS

Subtype:

Result: ALLOW

Config:

Additional Information:

Forward Flow based lookup yields rule:

in id=0x7fff38a10a50, priority=8, domain=conn-set, deny=false

hits=4069, user_data=0x7fff38770910, cs_id=0x0, reverse, flags=0x0, protocol=6

src ip/id=0.0.0.0, mask=0.0.0.0, port=0

dst ip/id=192.168.1.1, mask=255.255.255.255, port=443, dscp=0x0

input_ifc=wan, output_ifc=identity

Phase: 4

Type: IP-OPTIONS

Subtype:

Result: ALLOW

Config:

Additional Information:

Forward Flow based lookup yields rule:

in id=0x7fff395c7d70, priority=0, domain=inspect-ip-options, deny=true

hits=4044934, user_data=0x0, cs_id=0x0, reverse, flags=0x0, protocol=0

src ip/id=0.0.0.0, mask=0.0.0.0, port=0

dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, dscp=0x0

input_ifc=wan, output_ifc=any

Phase: 5

Type: VPN

Subtype: ipsec-tunnel-flow

Result: ALLOW

Config:

Additional Information:

Forward Flow based lookup yields rule:

in id=0x7fff37560700, priority=13, domain=ipsec-tunnel-flow, deny=true

hits=2268518, user_data=0x0, cs_id=0x0, flags=0x0, protocol=0

src ip/id=0.0.0.0, mask=0.0.0.0, port=0

dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, dscp=0x0

input_ifc=wan, output_ifc=any

Phase: 6

Type: TCP-MODULE

Subtype: webvpn

Result: ALLOW

Config:

Additional Information:

Forward Flow based lookup yields rule:

in id=0x7fff38a10cc0, priority=13, domain=soft-np-tcp-module, deny=false

hits=4627, user_data=0x7fff38c14300, cs_id=0x0, reverse, flags=0x0, protocol=6

src ip/id=0.0.0.0, mask=0.0.0.0, port=0

dst ip/id=192.168.1.1, mask=255.255.255.255, port=443, dscp=0x0

input_ifc=wan, output_ifc=identity

Phase: 7

Type: VPN

Subtype: encrypt

Result: DROP

Config:

Additional Information:

Reverse Flow based lookup yields rule:

out id=0x7fff375504a0, priority=69, domain=encrypt, deny=false

hits=40747, user_data=0x0, cs_id=0x7fff3754fa40, reverse, flags=0x0, protocol=0

src ip/id=192.168.1.1, mask=255.255.255.255, port=0

dst ip/id=10.0.0.1, mask=255.255.255.255, port=0, dscp=0x0

input_ifc=any, output_ifc=wan

Result:

input-interface: wan

input-status: up

input-line-status: up

output-interface: NP Identity Ifc

output-status: up

output-line-status: up

Action: drop

Drop-reason: (acl-drop) Flow is denied by configured rule

If I run packet-tracer with any other source IP address, let's say 10.0.0.2, everything is OK:

packet-tracer input wan tcp 10.0.0.2 50601 192.168.1.1 443 de

Phase: 1

Type: ROUTE-LOOKUP

Subtype: input

Result: ALLOW

Config:

Additional Information:

in 192.168.1.1 255.255.255.255 identity

Phase: 2

Type: ACCESS-LIST

Subtype:

Result: ALLOW

Config:

Implicit Rule

Additional Information:

Forward Flow based lookup yields rule:

in id=0x7fff37573f00, priority=119, domain=permit, deny=false

hits=862, user_data=0x0, cs_id=0x0, flags=0x0, protocol=6

src ip/id=0.0.0.0, mask=0.0.0.0, port=0

dst ip/id=0.0.0.0, mask=0.0.0.0, port=443, dscp=0x0

input_ifc=wan, output_ifc=identity

Phase: 3

Type: CONN-SETTINGS

Subtype:

Result: ALLOW

Config:

Additional Information:

Forward Flow based lookup yields rule:

in id=0x7fff38a10a50, priority=8, domain=conn-set, deny=false

hits=4090, user_data=0x7fff38770910, cs_id=0x0, reverse, flags=0x0, protocol=6

src ip/id=0.0.0.0, mask=0.0.0.0, port=0

dst ip/id=192.168.1.1, mask=255.255.255.255, port=443, dscp=0x0

input_ifc=wan, output_ifc=identity

Phase: 4

Type: IP-OPTIONS

Subtype:

Result: ALLOW

Config:

Additional Information:

Forward Flow based lookup yields rule:

in id=0x7fff395c7d70, priority=0, domain=inspect-ip-options, deny=true

hits=4047886, user_data=0x0, cs_id=0x0, reverse, flags=0x0, protocol=0

src ip/id=0.0.0.0, mask=0.0.0.0, port=0

dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, dscp=0x0

input_ifc=wan, output_ifc=any

Phase: 5

Type: VPN

Subtype: ipsec-tunnel-flow

Result: ALLOW

Config:

Additional Information:

Forward Flow based lookup yields rule:

in id=0x7fff37560700, priority=13, domain=ipsec-tunnel-flow, deny=true

hits=2270040, user_data=0x0, cs_id=0x0, flags=0x0, protocol=0

src ip/id=0.0.0.0, mask=0.0.0.0, port=0

dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, dscp=0x0

input_ifc=wan, output_ifc=any

Phase: 6

Type: TCP-MODULE

Subtype: webvpn

Result: ALLOW

Config:

Additional Information:

Forward Flow based lookup yields rule:

in id=0x7fff38a10cc0, priority=13, domain=soft-np-tcp-module, deny=false

hits=4648, user_data=0x7fff38c14300, cs_id=0x0, reverse, flags=0x0, protocol=6

src ip/id=0.0.0.0, mask=0.0.0.0, port=0

dst ip/id=192.168.1.1, mask=255.255.255.255, port=443, dscp=0x0

input_ifc=wan, output_ifc=identity

Phase: 7

Type: USER-STATISTICS

Subtype: user-statistics

Result: ALLOW

Config:

Additional Information:

Reverse Flow based lookup yields rule:

out id=0x7fff3a1cc320, priority=0, domain=user-statistics, deny=false

hits=4902651, user_data=0x7fff3a0043c0, cs_id=0x0, reverse, flags=0x0, protocol=0

src ip/id=0.0.0.0, mask=0.0.0.0, port=0

dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, dscp=0x0

input_ifc=any, output_ifc=wan

Phase: 8

Type: FLOW-CREATION

Subtype:

Result: ALLOW

Config:

Additional Information:

New flow created with id 4384689, packet dispatched to next module

Module information for forward flow ...

snp_fp_tracer_drop

snp_fp_inspect_ip_options

snp_fp_tcp_normalizer

snp_fp_tcp_mod

snp_fp_adjacency

snp_fp_fragment

snp_fp_drop

Module information for reverse flow ...

snp_fp_tracer_drop

snp_fp_inspect_ip_options

snp_fp_tcp_normalizer

snp_fp_adjacency

snp_fp_fragment

snp_ifc_stat

Result:

input-interface: wan

input-status: up

input-line-status: up

output-interface: NP Identity Ifc

output-status: up

output-line-status: up

Action: allow

I run packet capture on WAN interface - and I can only see incoming packets (SYN) with destination to tcp/443 but there isn't any outgoing packet (SYN/ACK).

I even can't open web page from internet browser (url https://192.168.1.1) when source IP is 10.0.0.1, but I can open "SSL VPN Service" web page from any other source IP address.

The only thing different with this IP address is that there's configured site-to-site (IPsec) vpn tunnel from same source to same destination IP address.

Here is the configuration of the tunnel:

group-policy GroupPolicy_10.0.0.1 internal

group-policy GroupPolicy_10.0.0.1 attributes

vpn-filter value VPN-ACL

vpn-tunnel-protocol ikev1 ssl-client

access-list VPN-ACL:

access-list VPN-ACL extended permit ip object-group DM_INLINE_NETWORK_83 object-group DM_INLINE_NETWORK_84

object-group network DM_INLINE_NETWORK_83

network-object 10.11.217.0 255.255.255.0

network-object 192.168.201.0 255.255.255.0

object-group network DM_INLINE_NETWORK_84

network-object 10.11.217.0 255.255.255.0

network-object 192.168.201.0 255.255.255.0

tunnel local & remote networks:

access-list wan_cryptomap_5 extended permit ip 10.11.217.0 255.255.255.0 192.168.201.0 255.255.255.0

crypto map wan_map 5 match address wan_cryptomap_5

crypto map wan_map 5 set connection-type answer-only

crypto map wan_map 5 set peer 10.0.0.1

crypto map wan_map 5 set ikev1 transform-set ESP-3DES-SHA

I've configured the same setup in my lab and I can't reproduce the error.

The SW version running on ASA is asa861-12.

I'm out of ideas.

Jernej Vodopivec · ‎11-28-2013

Hi,

I checked that there isn't any NAT translation happening for this traffic.

When running traffic capture on WAN interface I can only see 4 packets coming

- 3x SYN from remote IP

- 1x RST from remote IP

There wasn't any packet sent back from ASA to remote host.

Traffic filter (acl) for capturing this traffic is defined:

- IP; remote host 10.0.0.1, any

- ICMP; remot host 10.0.0.1, any

- IP; any, remote host 10.0.0.1

- ICMP; any, remote host 10.0.0.1

I can find this line in capture asp-drop type asp-drop: 14:26:49.865693 000b.dee6.4b00 30f7.8d47.9db4 0x0800 62: 10.0.0.1.51285 > 192.168.1.1.443: S [tcp sum ok] 3204589742:3204589742(0) win 8192 (DF) (ttl 124, id 17505)

Nothing else.

This is the first ACE in WAN's access list:

access-list wan_access_in line 1 extended permit ip host 10.0.0.1 any log emergencies interval 300

Still this is happening with only 1 remote host.

Does anyone have any idea how to further diagnose the problem?

Regards,

Jernej

Jernej Vodopivec · ‎11-28-2013

Just collected some other information:

1. traceroute shows that traffic is not leaving ASA at all

1 * * *

2 * * *

3 * * *

.......

I double checked that there is no "strange" entry for remote public IP in routing. Traffic with destination to remote IP should be sent via default gateway like all other traffic.

2. debug crypto ipsec shows this information when I ping public IP address of the remote host (with VPN

IPSEC(crypto_map_check)-3: Looking for crypto map matching 5-tuple: Prot=1, saddr=192.168.1.1, sport=30647, daddr=10.0.0.1, dport=30647

IPSEC(crypto_map_check)-5: Checking crypto map wan_map 1: skipping because 5-tuple does not match ACL wan_cryptomap_1.

IPSEC(crypto_map_check)-5: Checking crypto map wan_map 2: skipping because 5-tuple does not match ACL wan_cryptomap_2.

IPSEC(crypto_map_check)-5: Checking crypto map wan_map 3: skipping because 5-tuple does not match ACL wan_cryptomap_3.

IPSEC(crypto_map_check)-5: Checking crypto map wan_map 4: skipping because 5-tuple does not match ACL wan_cryptomap_4.

IPSEC(crypto_map_check)-5: Checking crypto map wan_map 5: skipping dormant map.

IPSEC(crypto_map_check)-5: Checking crypto map wan_map 6: skipping because 5-tuple does not match ACL wan_cryptomap_6.

IPSEC(crypto_map_check)-5: Checking crypto map wan_map 7: skipping because 5-tuple does not match ACL wan_cryptomap_7.

IPSEC(crypto_map_check)-5: Checking crypto map wan_map 8: skipping because 5-tuple does not match ACL wan_cryptomap_8.

IPSEC(crypto_map_check)-5: Checking crypto map wan_map 9: skipping because 5-tuple does not match ACL wan_cryptomap_9.

IPSEC(crypto_map_check)-5: Checking crypto map wan_map 10: skipping because 5-tuple does not match ACL wan_cryptomap_10.

IPSEC(crypto_map_check)-5: Checking crypto map wan_map 11: skipping because 5-tuple does not match ACL wan_cryptomap_11.

IPSEC(crypto_map_check)-5: Checking crypto map wan_map 13: skipping because 5-tuple does not match ACL wan_cryptomap_13.

IPSEC(crypto_map_check)-5: Checking crypto map wan_map 65535: skipping dynamic_link.

IPSEC(crypto_map_check)-1: Error: No crypto map matched.

It really seems that the whole problem is that ASA is trying to encrypt traffic sent from public IP address of one VPN endpoint and targeted to public IP address of another VPN endpoint and send it to remote VPN endpoint via IPcec tunel.

There is indeed VPN tunnel established between both VPN endpoints, but there are just local and remote networks defined with private IP address space for this tunnel, VPN endpoint's public IP addresses are not included in the definition of this IPsec VPN tunnel.

And there are at least two more IPsec VPN tunnels configured the same way and I can't reprodure this error on there two VPN tunnels.

Any idea?

m.kafka · ‎11-29-2013

I don't understand why you have a crypto map entry (IPsec) for 10.0.0.1 if this client should connect via SSL. Also: according to documentation the Anyconnect client doesn't support IKEv1, only IKEv2 can be used for Anyconnect.

Jernej Vodopivec · ‎11-29-2013

Hi,

I have crypto map entry for 10.0.0.1 because there is site-to-site VPN tunnel established between both locations: for servers to communicate with each other in both data centers.

But additionaly to this site-to-site IPsec VPN tunnel there are some people at remote location that want to establish SSL VPN remote connection that grants them additional access.

I would also like to mention: this setup was working fine until the problem occured about one week ago. The only thing that I'm aware of is that one week ago there was ASA's firmware updated from 8.6.1-2 to 8.6.1-12. But even when I downgrade firmware back to 8.6.1-2 the issue is still here.

ASA is encrypting traffic targeted to remote WAN IP address, even if it is TCP or ICMP traffic.

ikve1 debug shows: [IKE COMMON DEBUG]Tunnel Manager failed to dispatch a KEY_ACQUIRE message. Probable mis-configuration of the crypto map or tunnel-group. Map Tag = Unknown. Map Sequence Number = 0.

But the tunnel group for IPsec tunnel only includes remote and local network.

Jernej Vodopivec · ‎11-29-2013

The problem was in ipsec vpn tunel type: when it was changed from respond only to bidirectional, it started working.
Is this a bug or am I missing something?

m.kafka · ‎11-29-2013

Honestly, I can't give an answer whether its a bug or by design.

To my understaning it shouldn't have an effect for incoming connections.

you can read on

http://www.cisco.com/en/US/products/hw/vpndevc/ps2030/products_tech_note09186a0080b8ad03.shtml#asarespond

"This document provides information on how to configure a VPN gateway device to always act as a responder in an IKE negotiation. The device will respond to any crypto negotiations initiated by its peers."

So it seems to me a software bug...

Jernej Vodopivec · ‎11-30-2013

Is there any way to submit bug report to help Cisco create patch for this bug in next ASA software versions (ASA is not covered by Smartnet but I don't need any service from TAC center, just want to help improve software)?