AnyConnect to site-to-site: ICMP ok, TCP/UDP not; works OK with IPSec client.

[Paolo Marchiori]

Hi all,

see subject - AnyConnect client can't connect (TCP/22 aka SSH) to a server on the other end of IPSec site-to-site tunnel. 

Split tunnel is correctly configured, NAT exemption is OK, ACL (via DAP) is wide open. Routing works, intra-interface route is OK, etc.

In fact ping works perfectly - and logging shows ICMP buildup and teardown and even server can connect to the client.

But if I try TCP no packets arrive to the server, and logging shows nothing.

As if this wasn't funny enough, if the client does VPN using IPSec client (same group policy, same SAP) instead of AnyConnect, everything works.

Any suggestion?

[Marvin Rhoads]

Without seeing the configurations, have you tried packet tracer and/or packet capture to observe the traffic?

When you introduce the ssh attempt as interesting traffic do the encaps and decaps at each end of the IPsec SA increment as expected? (show crypto ipsec sa)

[Paolo Marchiori]

Packet tracer: the packet is allowed; IPsec SA counters are not incremented (this is kinda expected, as AnyConnect isn't ipsec and I think it's established that the problem is that packets form AC don't enter the ASA).

[Marvin Rhoads]

That's really odd. Routing must be working OK or else the icmp packets wouldn't make it.

Do I understand the topology correctly? I was assuming:

1. remote access AC clients come to ASA #1 via outside (Internet)
2. site-site VPN is from ASA #1 to ASA #2, also via outside interface
3. So RA AC client traffic has to hairpin at ASA #1.

The above works fine for AC clients' ICMP traffic but not for TCP.

IPsec remote access VPN clients into the same ASA#1 can connect to the remote site server just fine.

Are the AC and IPsec clients using the same address pool?

Are there any service policies that might be inspecting the AC clients' TCP traffic? (Though packet-tracer should catch that.)

The other thing that comes to mind is if there is any asymmetric routing in your network, there is a possibility the ASA may not see the whole 3-way handshake. That would explain icmp working while tcp fails. There's an example of that scenario in this blog post. (Tip of the hat to Matt O.)

[Paolo Marchiori]

1-2-3: correct. As both your other statements are.

The address pool is absolutely the same (a /23 network). No service policies that impact just the AC traffic. DAP catches both IKE and AC clients. No asymmetric routing (the network is very straightforward).

It's not just odd: it's driving me nuts. Actually it was driving me nuts even before I tried connecting thru IPsec client, which has only worsened the situation (and makes me think of some kind of bug in AC).

[ankit13389]

Hi,

Can you please let us know the ASA code which you are running on which Anyconnect client are terminating..

Also, take asp drop captures on ASA 1 on which clients are terminating:

--cap asp type asp-drop all

Run TCP traffic like RDP, SSH, Telnet etc and take the following output:

--sho asp drop | in <ip address of Anyconenct client>

[Paolo Marchiori]

Hi,

Cisco Adaptive Security Appliance Software Version 8.3(2)40 
Device Manager Version 7.1(4)

sho asp drop | i <client IP> yeld no results, but

cap asp type asp-drop all real-time 

GREPped for the client IP address gave me:

  34: 19:12:58.222949 10.172.85.215.52732 > 10.240.0.4.22: S 498261619:498261619(0) win 65535 <mss 1316,nop,wscale 4,nop,nop,timestamp 1036882611 0,sackOK,eol>
  56: 19:12:59.277680 10.172.85.215.52732 > 10.240.0.4.22: S 498261619:498261619(0) win 65535 <mss 1316,nop,wscale 4,nop,nop,timestamp 1036883681 0,sackOK,eol>
  83: 19:13:00.285355 10.172.85.215.52732 > 10.240.0.4.22: S 498261619:498261619(0) win 65535 <mss 1316,nop,wscale 4,nop,nop,timestamp 1036884688 0,sackOK,eol>
 109: 19:13:01.307997 10.172.85.215.52732 > 10.240.0.4.22: S 498261619:498261619(0) win 65535 <mss 1316,nop,wscale 4,nop,nop,timestamp 1036885695 0,sackOK,eol>
 156: 19:13:02.305816 10.172.85.215.52732 > 10.240.0.4.22: S 498261619:498261619(0) win 65535 <mss 1316,nop,wscale 4,nop,nop,timestamp 1036886702 0,sackOK,eol>
 181: 19:13:03.342969 10.172.85.215.52732 > 10.240.0.4.22: S 498261619:498261619(0) win 65535 <mss 1316,nop,wscale 4,nop,nop,timestamp 1036887711 0,sackOK,eol>

which reassures me on the fact that SYNs get in and don't get to the destination, but gives me no idea on why the packets are dropped.

[Marius Gunnerud]

Do the IPsec VPN and AC VPN receive IP addresses from the same address POOL?  If not, you might want to check that your NAT exempt and crypto ACLs are correct.

--

Please remember to select a correct answer and rate

[Marius Gunnerud]

Also would be helpful to see the full (sanitised) configuration of both ASAs if possible.

--

Please remember to select a correct answer and rate

[Paolo Marchiori]

The ASA to which the AC clients connect has a 395-Kb configuration; the remote ASA is not under my control - I actually doubt it's even an ASA...

[Paolo Marchiori]

As per my previous reply, it's the same pool, same group policy, same DAP.

[Paolo Marchiori]

In case someone hits this post by searching: it was due to bugs CSCty32412 and CSCtz48136 (have no access to the second, but the first is pretty much it).

Updating ASA to 8.4.7 solved the problem.

[Marvin Rhoads]

Thanks for letting us know the resolution. Cheers!