UC500 and IPsec VPN clients - disconnects

[JP Babiera]

Just throwing a question out there.
I have a UC560 running uc500-advipservicesk9-mz.151-2.T2 at an HQ site.  Remote users, about 8 of them, are trying to connect via IPsec VPN clients (v5.0.07.0440) to HQ to access files, etc.  The behavior I'm seeing is that 5 users successfully connect, but only 5.  As soon as more users trying to connect, they either:

1. connect successfully for a min, then drop
2. get a 412, Remote peer is no longer responding
3. connect, but kick someone else's session off.

The users are using the same VPN profile, but with unique usernames and passwords.

Here's some of the UC configs for VPN clients
crypto isakmp client configuration group USER01
 key ********
 dns 192.168.0.110
 pool USER01_POOL
 acl USER01_ACL

aaa authentication login RAUTHEN local
aaa authorization network RAUTHOR local if-authenticated

crypto isakmp profile USER01_PROF
   match identity group USER01
   client authentication list RAUTHEN
   isakmp authorization list RAUTHOR
   client configuration address respond

crypto isakmp policy 1
 encr 3des
 hash md5
 authentication pre-share
 group 2
crypto isakmp policy 10
 encr aes
 authentication pre-share
 group 2
 lifetime 28800
crypto isakmp policy 100
 encr aes
 authentication pre-share
 group 2
 lifetime 3600
crypto isakmp policy 1000
 encr 3des
 authentication pre-share
 group 2

I enabled debugs
debug crypto isakmp
debug crypto ipsec

This is some of the things I see on the debugs
604899: Aug 21 16:41:13.333: ISAKMP:(2073): processing HASH payload. message ID = 284724149
604900: Aug 21 16:41:13.333: ISAKMP:(2073): processing NOTIFY DPD/R_U_THERE protocol 1
        spi 0, message ID = 284724149, sa = 0x8E7C6E68
604901: Aug 21 16:41:13.333: ISAKMP:(2073):deleting node 284724149 error FALSE reason "Informational (in) state 1"
604902: Aug 21 16:41:13.333: ISAKMP:(2073):Input = IKE_MESG_FROM_PEER, IKE_INFO_NOTIFY
604903: Aug 21 16:41:13.333: ISAKMP:(2073):Old State = IKE_P1_COMPLETE  New State = IKE_P1_COMPLETE 

581504: Aug 20 16:59:12.805: ISAKMP:(2147):purging node -1455244451
581505: Aug 20 16:59:12.805: ISAKMP:(2147):purging node 840814618
581506: Aug 20 16:59:13.933: ISAKMP (2147): received packet from 201.195.231.162 dport 4500 sport 37897 Global (R) QM_IDLE      
581507: Aug 20 16:59:13.933: ISAKMP: set new node 801982813 to QM_IDLE      
581508: Aug 20 16:59:13.933: ISAKMP:(2147): processing HASH payload. message ID = 801982813
581509: Aug 20 16:59:13.933: ISAKMP:received payload type 18
581510: Aug 20 16:59:13.933: ISAKMP:(2147):Processing delete with reason payload
581511: Aug 20 16:59:13.933: ISAKMP:(2147):delete doi = 0
581512: Aug 20 16:59:13.933: ISAKMP:(2147):delete protocol id = 1
581513: Aug 20 16:59:13.933: ISAKMP:(2147):delete spi_size =  16
581514: Aug 20 16:59:13.933: ISAKMP:(2147):delete num spis = 1
581515: Aug 20 16:59:13.933: ISAKMP:(2147):delete_reason = 2
581516: Aug 20 16:59:13.933: ISAKMP:(2147): processing DELETE_WITH_REASON payload, message ID = 801982813, reason: DELETE_BY_USER_COMMAND
581517: Aug 20 16:59:13.933: ISAKMP:(2147):peer does not do paranoid keepalives.

581518: Aug 20 16:59:13.933: ISAKMP:(2147):peer does not do paranoid keepalives.

581519: Aug 20 16:59:13.933: ISAKMP:(2147):deleting SA reason "BY user command" state (R) QM_IDLE       (peer 201.195.231.162)
581520: Aug 20 16:59:13.933: ISAKMP:(2147):deleting node 801982813 error FALSE reason "Informational (in) state 1"
581521: Aug 20 16:59:13.933: ISAKMP: set new node -878597687 to QM_IDLE      
581522: Aug 20 16:59:13.937: ISAKMP:(2147): sending packet to 201.195.231.162 my_port 4500 peer_port 37897 (R) QM_IDLE      
581523: Aug 20 16:59:13.937: ISAKMP:(2147):Sending an IKE IPv4 Packet.
581524: Aug 20 16:59:13.937: ISAKMP:(2147):purging node -878597687
581525: Aug 20 16:59:13.937: ISAKMP:(2147):Input = IKE_MESG_INTERNAL, IKE_PHASE1_DEL
581526: Aug 20 16:59:13.937: ISAKMP:(2147):Old State = IKE_P1_COMPLETE  New State = IKE_DEST_SA 

I've opened a case with TAC on this and they can't seem to figure out what is causing this.  To them it seems like an undocumented bug.  And their recommendation is to reboot, upgrade, or try configuring L2TP for those remote users.

Thanks,

JP

[Frank DeNofa]

JP,

From the debugs we can see that we are receiving a DELETE message from the client at 201.195.231.162:

581506: Aug 20 16:59:13.933: ISAKMP (2147): received packet from 201.195.231.162 dport 4500 sport 37897 Global (R) QM_IDLE      
581507: Aug 20 16:59:13.933: ISAKMP: set new node 801982813 to QM_IDLE      
581508: Aug 20 16:59:13.933: ISAKMP:(2147): processing HASH payload. message ID = 801982813
581509: Aug 20 16:59:13.933: ISAKMP:received payload type 18
581510: Aug 20 16:59:13.933: ISAKMP:(2147):Processing delete with reason payload
581511: Aug 20 16:59:13.933: ISAKMP:(2147):delete doi = 0
581512: Aug 20 16:59:13.933: ISAKMP:(2147):delete protocol id = 1
581513: Aug 20 16:59:13.933: ISAKMP:(2147):delete spi_size =  16
581514: Aug 20 16:59:13.933: ISAKMP:(2147):delete num spis = 1
581515: Aug 20 16:59:13.933: ISAKMP:(2147):delete_reason = 2
581516: Aug 20 16:59:13.933: ISAKMP:(2147): processing DELETE_WITH_REASON payload, message ID = 801982813, reason: DELETE_BY_USER_COMMAND

Typically, you will see this when the user terminates the connection. If this is not the case, it is most likely something with the client causing a delete to be sent in error. Unfortunately, as the legacy IPsec VPN client is End of Support, TAC may not be able to provide you with a permanent fix. If you could text from a different client (Cisco iOS built-in client, Mac OSX built-in client, another Cisco router acting as a client, etc;), this could help confirm if the Cisco VPN Client is what is causing the issue.

HTH,

Frank

[JP Babiera]

Thanks for the reply Frank.

We'll be trying an IOS upgrade on this UC560, but I have a feeling that this is just a limitation of that device.  Same users on the same Windows 7 PCs, running the same client, are able to connect to another UC500 at a different site with identical configurations.

I was wondering if I was running into a feature licensing issue on the UC560 as seen on http://www.cisco.com/c/en/us/products/collateral/unified-communications/unity-express/reference_guide_c07-566560.html  (table 5).  But we're not even hitting 20 VPN tunnels.  Unless the device thinks that when 5 users from the same source IP, it counts as the 5 teleworkers per site.

If the upgrade doesn't work, we'll have to look at other means to get this done.

Thanks,

JP

[Frank DeNofa · Accepted solution]

JP,

An IOS upgrade is worth a shot, although the debugs seem to indicate it is an issue with the client. If possible, I would still suggest testing with another client to see if it's unique to the Cisco VPN Client on Win7. Regarding the 20 tunnel limit, this most likely refers to the number of IPsec SAs. If you issue a "show crypto eli," this will print the number of IPSec-Sessions which are currently active.

HTH,

Frank

[JP Babiera]

Ah!  i think I'm reaching the maximum IPsec sessions that this router is capable of.

Router#sh cry eli
Hardware Encryption : ACTIVE
 Number of hardware crypto engines = 1

 CryptoEngine Onboard VPN details: state = Active
 Capability    : IPPCP, DES, 3DES, AES, IPv6, GDOI, FAILCLOSE

 IPSec-Session :    56 active,    60 max, 5 failed

[Frank DeNofa · Accepted solution]

That looks like that'll do it. Keep in mind that each IPsec "tunnel" (ie. client connection) will have an inbound and outbound SPI, each of which count as an "IPSec-Session" in this "show crypto eli" output. Therefore, the 60 max session equates to 30 client connections.

Frank

[JP Babiera]

Thanks a lot for the help Frank!  That show output helped point me in the right direction on this one.