Problems with VPN disconnections after upgrading ASA to address Cisco security advisory
So, as some background, we have been happily running ASA 8.4(3) for some time. But we actually first experienced this problem a while ago when trying to upgrade ASA software version but downgraded back to 8.4(3) again, the problem went away, and it got sidetracked for various reasons (primarily that we intended upgrading the ASA hardware anyway though, typically, that hasn't actually been carried out yet!)
Anyway, following the latest Cisco security advisory:
BUT - the same problems we experienced last time we tried upgrading (on that occasion it was to 8.4.6) were experienced again. Basically, after upgrading, we experience VPN connectivity problems with (at least) one of our sites - the VPN connectivity with them constantly establishes then drops, then re-establishes again. If we return to 8.4(3) everything is fine.
After upgrading, on the Standby ASA, I notice the following messages constantly appearing:
Apr 22 2014 17:27:16: %ASA-3-210005: LU allocate connection failed Apr 22 2014 17:27:16: %ASA-3-210005: LU allocate connection failed Apr 22 2014 17:27:16: %ASA-3-210005: LU allocate connection failed Apr 22 2014 17:27:16: %ASA-3-210005: LU allocate connection failed Apr 22 2014 17:27:17: %ASA-3-210005: LU allocate connection failed Apr 22 2014 17:27:18: %ASA-3-210005: LU allocate connection failed Apr 22 2014 17:27:19: %ASA-3-210005: LU allocate connection failed
When the Standby ASA is made Active (as the "Old Active" is being updated) we then get the following messages for one particular site (we also have several other sites which didn't appear to be affected):
Apr 22 2014 17:27:31: %ASA-3-713061: Group = 220.127.116.11, IP = 18.104.22.168, Rejecting IPSec tunnel: no matching crypto map entry for remote proxy 192.168.1.0/255.255.255.0/0/0 local proxy 22.214.171.124/255.255.255.255/0/0 on interface OUTSIDE Apr 22 2014 17:27:31: %ASA-3-713902: Group = 126.96.36.199, IP = 188.8.131.52, QM FSM error (P2 struct &0x72f2a728, mess id 0xf2924361)! Apr 22 2014 17:27:31: %ASA-3-713902: Group = 184.108.40.206, IP = 220.127.116.11, Removing peer from correlator table failed, no match! Apr 22 2014 17:27:31: %ASA-3-752015: Tunnel Manager has failed to establish an L2L SA. All configured IKE versions failed to establish the tunnel. Map Tag= outside_map. Map Sequence Number = 3002.
Now, this relates to something of a peculiarity we have in our set-up. At the remote site, we split-tunnel internet traffic EXCEPT for traffic to one particular "extranet" site which is routed back through the datacentre ASA and out to the internet through the DC internet connection. (This all works fine on 8.4(3) though)
This makes me think its due to some sort of mismatch between both ends of the connection but, if so, why does it work ok on version 8.4(3)? And the other strange thing is that we have another remote site almost identically configured which doesn't appear to suffer any such problems. The remote site has a Cisco 2911 running 15.1(4)M.
At the time of the issue, the only messages appearing on the Cisco 2911 were:
Apr 22 17:27:31.238: %CRYPTO-4-RECVD_PKT_INV_SPI: decaps: rec'd IPSEC packet has invalid spi for destaddr=18.104.22.168, prot=50, spi=0x95A15F88(2510380936), srcaddr=22.214.171.124, input interface=GigabitEthernet0/0 Apr 22 17:28:31.250: %CRYPTO-4-RECVD_PKT_INV_SPI: decaps: rec'd IPSEC packet has invalid spi for destaddr=126.96.36.199, prot=50, spi=0x4008F420(1074328608), srcaddr=188.8.131.52, input interface=GigabitEthernet0/0 Apr 22 17:32:59.554: %CRYPTO-4-RECVD_PKT_INV_SPI: decaps: rec'd IPSEC packet has invalid spi for destaddr=184.108.40.206, prot=50, spi=0xA198CD52(2711145810), srcaddr=220.127.116.11, input interface=GigabitEthernet0/0
Datacentre - ASA5520 running version 8.4(3), all works fine. If upgraded to e.g. 8.4(6) or 8.4(7.15) then problems occur but only seems to affect one particular site
Remote Site (Office101) - Cisco 2911 running 15.1(4)M
crypto isakmp key **** address 18.104.22.168 ! crypto map cryptomap 10 ipsec-isakmp set peer 22.214.171.124 set transform-set vpn-set match address datacentre ! ip access-list extended datacentre permit ip 192.168.1.0 0.0.0.255 10.10.0.0 0.0.255.255 permit ip 192.168.1.0 0.0.0.255 172.16.11.48 0.0.0.15 permit ip 192.168.1.0 0.0.0.255 172.16.10.0 0.0.0.255 permit ip 192.168.1.0 0.0.0.255 172.16.12.0 0.0.0.127 permit ip 192.168.1.0 0.0.0.255 host 126.96.36.199 ! !###NO NAT ACL#### access-list 112 deny ip 192.168.1.0 0.0.0.255 10.10.0.0 0.0.255.255 access-list 112 deny ip 192.168.1.0 0.0.0.255 172.16.10.0 0.0.0.255 access-list 112 deny ip 192.168.1.0 0.0.0.255 172.16.11.48 0.0.0.15 access-list 112 deny ip 192.168.1.0 0.0.0.255 172.16.12.0 0.0.0.127 access-list 112 deny ip 192.168.1.0 0.0.0.255 host 188.8.131.52 access-list 112 permit ip 192.168.1.0 0.0.0.255 host [cloud-web-scan-ip] access-list 112 deny ip any any
Has anyone come across similar and/or can suggest what the problem might be? It seems to suggest a misconfiguration but why does it work ok for 8.4(3)? I am concerned now that we cannot address any security vulnerabilities or suchlike because upgrading the ASA version seems to introduce this issue. Also, as I mentioned, we intend upgrading the ASA hardware (to ASA5545x) and, as such, would likely be running 8.6 software on this appliance but I'm worried that introducing the new hardware might also bring in this problem as a result, so I'm naturally keen to find the cause of the issue!
thanks for the response. The permit ip 192.168.1.0 0.0.0.255 host 184.108.40.206 line on the 2911 is to direct traffic to the extranet site (220.127.116.11) through the tunnel to the datacentre ASA (rather than split-tunneling through the 2911's internet connection) and through the datacentre internet connection. So the ACL containing that line is for the "interesting" traffic that will be sent across the VPN tunnel to the datacentre ASA. This seems to work fine with the ASA version 8.4(3)
However, no, I don't have the "sysopt connection preserve-vpn-flows" line - that is interesting, may actually be of benefit? But the thing that would still puzzle me would be why everything works fine at 8.4(3) but only when I upgrade do I experience issue?
it turned out the problem was due to a mismatch in the ACLs between the ASA and the remote site 2911.
We had no matching line in the ASA config for traffic between the office and the extranet server. On the office router, we had:
permit ip 192.168.1.0 0.0.0.255 host 18.104.22.168
So, on the ASA, we had to add:
access-list office101 line 8 extended permit ip host 22.214.171.124 192.168.1.0 255.255.255.0
I'm not sure why the behaviour was different between the ASA versions i.e. why we did not experience any issues when we had the misconfig at version 8.4(3) but it caused problems with higher versions. Also, cannot explain why it only really seemed to be one site that was affected (we had the same config mismatch at other sites which were seemingly alright)
However, ultimately, it was the config mismatch that was causing the problem and, after rectifying that, we were able to upgrade and our remote site VPN connectivity remained stable.
Table of ContentsIntroductionVersion HistoryPossible Future
UpdatesDocuments PurposeNAT Operation in ASA 8.3+ SectionsRule Types
Network Object NATTwice NAT / Manual NATRule Types used per SectionNAT
Types used with Twice NAT / Manual NAT and Network Obje...
Table of Contents Introduction:This document describes details on how
NAT-T works. Background: ESP encrypts all critical information,
encapsulating the entire inner TCP/UDP datagram within an ESP header.
ESP is an IP protocol in the same sense that TCP an...