Split tunnel works... but only for one IP

[justinmm2]

Hi All,

Dealing with a really frustrating problem. Our setup, roughly, is as follows:

- We have a remote access VPN that users connect to with any connect; in turn, they are assigned a local LAN address: 10.1.11.192-10.1.11.200

- We have a site-to-site VPN that connects to Amazon AWS to access 10.0.249.0 and other subnets, and now certain hosts on the Amazon *public* network (e.g, 54.1.2.3). This is done via a split tunnel.

What we're seeing is this:

- Users connect to the VPN and are assigned one of the addresses above. Let's use 10.1.11.192 for this example.

- They can then access anything in the 10.0.249.0 subnet (via the split tunnel) just fine. This goes across both ASA devices.

- They can then access anything in the public Amazon network (via the split tunnel) just fine. This should only use the remote access ASA.

So, it seemed like everything was working. When connected to the VPN, Amazon hosts in both 10.x.x.x networks and public IPs that I had specifically tunneled (we plan to transition these to a VPC soon) were accessible, and access happened via the remote access VPN IP (i.e, when connecting to 54.1.2.3, it showed the user being logged in from the Cisco's gateway's IP address, as opposed to the local client IP).

Now, here's where things got weird: *public* tunneled hosts at Amazon only works with the first address in the pool, 10.1.11.192. No other addresses work. 10.0.249.x is always available, regardless of assigned IP. 54.x.y.z is only available with .192.

I've used the same computer with different assigned IPs (10.1.11.193-10.1.11.200), and none work. I've connected using different computers.. they work if assigned .192, but not any other addresses. Other users report the same issue.

TCP handshaking is failing

I'll use our IRC server (and sometimes ssh server) for testing. I can see my client laptop with a SYN_SENT on that specific connection. I can see the IRC server with a SYN_RECV, and the ASA shows a SYN timeout after 30 seconds. Thus, it looks as though packets from the IRC server can't make their way back through the ASA to my client laptop.

I suspect this has something to do with dynamic vs. static NAT, etc., but I've fiddled with every setting I can, and coming up blank.

I'm also baffled as to why .192 works, but no other addresses do.

I've attached our configuration, minus keys and passwords and certain IPs/hostnames. It's a little bit ugly because it has some poor attempts at fixing this, things I'll probably remove after it works, but.. Could it have anything to do with TCP sequence randomization?

Thank you in advance for any help.

[Jouni Forss]

Hi,

After a long day at work my head is a bit slow so I'll start lightly for my own sake

Did I understand correctly that you use Split Tunnel and the purpose of the Split Tunnel is to both only tunnel local networks and also tunnel some selected public IP address through the VPN connection and to the ASA? And these public destination IP addresses are connected through the VPN connection so that the users  can utilize the public IP address configured on the ASA "outside" interface?

Are all of the destination hosts/networks located behind a L2L VPN connection or were some connection actually being routed through the Internet connection directly to their public destination IP address? I can't see any L2L VPN configurations in the attached configuration file.

I would personally suggest "easing up" with the use of ACLs There is ACL for both interface in both direction and a global ACL also. I tend to go with an inbound ACL per interface. I rarely have a need to use any other direction with ACL or a global ACL for that matter.

I would also suggest having another VPN Pool network so that there is absolutely no overlap with the LAN and VPN networks.

After we confirm the above things we could take some "packet-tracer" output of the successfull connections and failed connections.

- Jouni

[justinmm2]

Hi Jouni! Thank you so much for your quick reply. Mine was delayed because I've continued to fight the ASA this afternoon! Apologies for my verbosity, always found it better to include too much info than too little!

To clarify, ideally, a VPN client that connects to this ASA (10.1.11.5) should have three possible destinations:

- The internet at large - this should not go through the ASA, but should go through the client's own gateway, bypassing this VPN entirely. This works.

- My servers in an Amazon VPC (10.0.249.x and some others) - this should come in to this ASA (10.1.11.5), which in turn is already successfully routed to another ASA (10.1.11.4 in the config). This works, too.

- My servers at Amazon AWS that are on the public internet (example below: 107.22.xxx.yyy). This should only involve the one ASA in question (10.1.11.5). This is where I'm running into issues.

The split tunnel, thus, includes the networks 10.0.249.x and specific public AWS hosts like 107.22.xxx.yyy, etc. I want 10.0.249.x to go to Amazon via the other ASA 10.1.11.4 (again, this works) and I want 107.22.xxx.yyy, etc. to go to Amazon via *this* ASA (10.1.11.5). Basically, it's this last part that's causing problems: it works only one in very, very narrow situation: when the client is assigned the address 10.1.11.192. If a user logs in and is assigned any other address, they won't be able to access the "public" AWS servers.

I began by taking your advice and moving the IP assignment to an entirely different subnet: 10.1.12.100-10.1.12.200. I then spent a decent chunk of the afternoon adjusting NAT rules (and removing ACLs, per your suggestion). The only brief success I had was when I had about a bajillion NAT rules, and somehow I made it on to the IRC server! Oddly, my connection used the ASA's address itself, rather than the ASA gateway's address. Unfortunately, I stopped being able to reach 10.0.249.x at the time :/ I tried to adjust further, but cannot restore/recall what the NAT rules were.

I'm miffed because 10.1.11.192 works, and *nothing* else does. There is nothing special about this address in the config - I've tried everything I can to figure out why this one address is different. I've also looked in places that *shouldn't* matter because this bypasses the VPC entirely (AWS security groups, AWS iptables rules, VPC options, etc.)

Again, the behavior of 10.1.11.193-200 is a partial TCP handshake: the local computer sees SYN_SENT, and the remote server has SYN_RECV, but no connection is ultimately established. This is why I think it could be a NAT issue, but I'm obviously a bit lost here Here's a sample connection attempt, with DNS traffic removed:

2013-10-17 18:29:09.100 [DEBUG] Message from Host: 10.1.11.5 - Message - %ASA-6-302013: Built inbound TCP connection 43606 for outside:10.1.11.193/56626 (10.1.11.193/56626)(LOCAL\justinsTestMac) to outside:107.22.xxx.yyy/6667 (107.22.xxx.yyy/6667) (justinsTestMac) (pid:25912)

2013-10-17 18:29:39.129 [DEBUG] Message from Host: 10.1.11.5 - Message - %ASA-6-302014: Teardown TCP connection 43606 for outside:10.1.11.193/56626(LOCAL\justinsTestMac) to outside:107.22.xxx.yyy/6667 duration 0:00:30 bytes 0 SYN Timeout (justinsTestMac) (pid:25912)

2013-10-17 18:29:45.293 [DEBUG] Message from Host: 10.1.11.5 - Message - %ASA-6-302013: Built inbound TCP connection 43612 for outside:10.1.11.193/56626 (10.1.11.193/56626)(LOCAL\justinsTestMac) to outside:107.22.xxx.yyy/6667 (107.22.xxx.yyy/6667) (justinsTestMac) (pid:25912)

102 (justinsTestMac) (pid:25912)

2013-10-17 18:30:15.322 [DEBUG] Message from Host: 10.1.11.5 - Message - %ASA-6-302014: Teardown TCP connection 43612 for outside:10.1.11.193/56626(LOCAL\justinsTestMac) to outside:107.22.xxx.yyy/6667 duration 0:00:30 bytes 0 SYN Timeout (justinsTestMac) (pid:25912)

2013-10-17 18:30:17.976 [DEBUG] Message from Host: 10.1.11.5 - Message - %ASA-6-302013: Built inbound TCP connection 43617 for outside:10.1.11.193/56626 (10.1.11.193/56626)(LOCAL\justinsTestMac) to outside:107.22.xxx.yyy/6667 (107.22.xxx.yyy/6667) (justinsTestMac) (pid:25912)

2013-10-17 18:30:48.400 [DEBUG] Message from Host: 10.1.11.5 - Message - %ASA-6-302014: Teardown TCP connection 43617 for outside:10.1.11.193/56626(LOCAL\justinsTestMac) to outside:107.22.xxx.yyy/6667 duration 0:00:30 bytes 0 SYN Timeout (justinsTestMac) (pid:25912)

Thanks for any insight you have!

[Jouni Forss · Accepted solution]

Hi,

I also rather explain everything to the detail. Though sometimes it just too much for my head when I am tired

Did you manage to correct the problem that arised from changing the settings?

The "packet-tracer" output for the failing connection would be important.

But now that I look at your original configurations and consider that your VPN Clients need to get to a select public IP addresses through the ASA it seems to me that your problem might be lack of NAT configuration for this traffic. (which the "packet-tracer" might indicate)

You would need a Dynamic PAT from "outside" to "outside" for the VPN users to be PATed to the ASA external IP address

Something like this for example

object network VPN-CLIENT-AMAZON-AWS-PAT

subnet 10.1.12.0 255.255.255.0

nat (outside,outside) dynamic interface

Or if your original VPN Pool is in use, change the above network.

The purpose of the above Dynamic PAT configuration is essentially to catch the VPN traffic coming from behind "outside" which is heading back through the "outside" interface and applying Dynamic PAT to the public IP address of the ASA. At the moment it would seem to me that the 10-network address is going through the ASA without NAT which essentially leads to the SYN timeout logs.

Though if I understood correctly you are saying that one of the VPN pool addresses reaches the public destination IP address which again doesnt really match with the above situation. However I dont see any NAT/PAT configuration for the VPN traffic to the public IP address. Look at your logs messages. They mention the same VPN Pool IP address twice (other inside the () ) which essentially means there is no NAT for the source address and the ISP drops the traffic naturally.

- Jouni

[justinmm2]

Jouni,

     Thank you so much! I had tried variations on this nat rule, but I guess I'd never done it properly - as best I can tell so far, this seems to work! Thank you again! I am trying it on the 10.1.11.0 network, but may move it to the 10.1.12.0 soon - may be best to give VPN users their own subnet. I think part of my confusion may have stemmed from trying to do things in ASDM rather than the CLI... lesson learned: don't bother with GUIs!

Cheers!

Justin