Welcome to the Cisco Networking Professionals Ask the Expert conversation. This is an opportunity to discuss Configuring and Troubleshooting IPSec on VPN with Cisco expert Afaq Khan. Afaq is part of Technical Assistance Center (TAC) based out of San Jose, where he has been working for the VPN TAC team as the customer support engineer for almost two years now. Feel free to post any questions relating to Configuring and Troubleshooting IPSec on VPN. Remember to use the rating system to let Afaq know if youve received an adequate response.
Afaq might not be able to answer each question due to the volume expected during this event. Our moderators will post many of the unanswered questions in other discussion forums shortly after the event. This event lasts through June 27. Visit this forum often to view responses to your questions and the questions of other community members.
We have network that consist of two VPN3K Concentrator at the HQ as a Master and a Backup device for several branch offices`s VPN3002. The design is using EzVPN mode with RRI to inject the branch office network segment to the head-end Concentrator.
My concern is sometimes we are having problem with the fail-over function. Our testing with fail-over, if there is some routing problem to the Master VPN3000 at the head-end, the VPN3002 will retry the Backup Concentrator. But it takes as long as 5 minutes to 7 minutes until all the network convergence.
My questions :
1. Is it by design that the fail-over is extremely slow (I think, 5 minutes is considered too slow for a redundancy design).
2.What kind of routing protocol does the RRI use?? From the head-end Concentrator`s routing table, it shows that the route from VPN3002 is learned through RIP . Does it mean that RRI uses RIP protocol to advertised 3002`s route through the encrypted tunnel ???
3. If RRI is "RIP-like" protocol, does all the timers(hold-down, hello, etc) of RIP apply also to the RRI ?
Appreciate for your insight.
First let me understand, if I get your Q right,
Are you talking abt VRRP cluster or two concentrators configured as backup for each other independtly w/o any VRRP for remote clients?
I guess you're talking abt latter scenario, in that case:
We'd need to look into as to what we're seeing in the IKE/IKEDBG/IPSec/IPSecDBG logs on the vpn3k and vpn3002 to figure out the delay, but you need to keep in mind some caveats:
* VPN3002 tries each and every server with the (calculated from initial IKE packet sent) timeout value hardcoded as 8 sec(and as per my understanding it makes one retry, so that makes wait/attempt time 8*2 =16 sec for each backup), so it basically tries all the backup servers in that order with this timeout value.
* If the VPN 3002 cannot connect after trying all backup servers on the list, it does not automatically retry.
- In Network Extension mode, the VPN 3002 attempts a new connection after 4 seconds.
- In Client mode, the VPN 3002 attempts a new connection when the user clicks the Connect Now button on the Monitoring | System Status screen, or when data passes from the VPN 3002 to the VPN Concentrator.
1. Failover in case of VRRP should not take more than 10 seconds, ie backup assuming the role of the Master.
2. RRI doesn't use any routing protocol per-se, RRI causes only VPN3K to install/remove routes based on client/vpn3002 connections/disconnection, vpn3k would n't advertise these routes unless you configure it with RIP or OSPF (supported routing protocols on VPN3K) to the inside devices.
3. Again, RRI is not tied up with any routing protocol, it depends on what you configure the vpn3k with, your all inside L3 devices will converge as per that routing protocol that you choose on VPN3K for RRI(OSPF for example would provide much faster convergence for RRI routes vis-a-vis RIP).
if you're seeing different behaviour, or cause of delay, feel free to open up TAC case to follow up on that.
This is regarding Concentrator version 3.6.7F annoying problem. I have customer with VRRP and they have clients that connected using IPSec over TCP. For some reasons (the client`s environment problem) they are unable to connect using IPSec over UDP. Testing with Concentrator version 3.6.7 F with VRRP redundancy has some issue with IPSec over TCP client. If the VRRP Master is failed , the Backup will become the Master, and at this time the BUG kicks in. The Master will not accept any IPSec over TCP clients. My question is what are your comment about this issue ? Cisco recommends to upgrade all version from v3.5.x , v3.6.x to "version 3.6.7F" because of the vulnerability of the Concentrator software (ICMP attack, SSH problem, IPSec over TCP hacking), but this version has issue with VRRP. Recently the version 3.6.7G is already released and there is no mention about fixing the VRRP bug issue. Do you have any recommendation how to solve my customer problem ???
Really appreciate for any insight.
IPsec over TCP/VRRP issue is documented as CSCeb22460, and its already resolved, and incorporated in 4.0.1B VPN3K code, this code is not posted on CCO yet, but should be there soon.
This is the follow up message for this issue, VPN3K OS V4.0.1B is posted now on CCO.
Our testing with the current redundancy solution (VRRP, Load-balance, Master&Backup in EzVPN) , if one of the device failed, the VPN clients has to disconnect the connection and re-connect again. The fail-over is not transparent to the VPN client. Is there any consideration to implement VPN Client "IPSec statefull failover" with VPN3000 ?
There is no stateful IPSec failover planned for VPN3K as of now.
As of now, only IOS supports stateful IPSec failover via SSP.
There is no stateful IPSec failover planned for VPN3K as of now.
Only IOS supports stateful IPSec failover to date.
This regarding VPN3002 buffering issue. One of our network having perfomance degradation issue with VPN3002 if the PCs behind it are transferring large FTP file or Lotus Notes syncing traffic. This VPN3002 is connected to the Provider using PPPoE. Lowering the MTU size to 1300 bytes only help a little bit. It doesn`t resolve the issue. From the Monitoring->MIB II -> IP -> Assembly failures counter, while transferring an 40MB FTP file , at some point the FTP will be aborted, and the counter will show a large number of assembly failure. My questions :
1. Does VPN3002 so weak for assembling process ?
2. How big is the buffer allocated for packet processing ? Do you have any number ? Does VPN3000 has any hidden command to show the buffer usage ( something like "show buffer").
3. Do you have any insight if we use PIX 506 (much cheaper than VPN3002) will resolve the assembly issue ?
Sorry for having many questions, really appreciate if you can help us troubleshooting the VPN3000 issues.
I have a question to the Backup LAN-to-LAN feature. Does ist work with PIX or router on the remote site or only with VPN3000.
Assembly failure refers to "The number of failures detected by the IP reassembly algorithm (for whatever reason: timed out, errors, etc.). This number is not necessarily a count of discarded IP fragments since some algorithms can lose track of the number of fragments by combining them as they are received"
If you are seeing a lots of failures, it could be because of the large IP packets being pushed across(could be near MTU size packets) from behind the headend, so the headend device basically fragmenting them after encryption (default method on VPN3000 concentrator), in that case your VPN3002 will have to re-assemble those packets prior to decryption, which is always a cpu intensive process and can bring down your box to its knees, eventually causing timeout/failures.
My suggestion would be to select "Fragment prior to IPSec encapsulation without Path MTU Discovery (Clear DF bit) " on the headend(VPN 3000 concentrator) for its Public interface, so that VPN3002 doesn't have to do reassembly at all rather end stations do that.
1. No its not, but packet reassembly is always a cpu intensive process for any device.
2 . There is no hidden command for that, it should be failr large interface buffer, as VPN3002 as 16MB of DRAM for CPU/IO.
3. I dont think, pix506 would do any better, if you continue to have fragmented IPSec packets sent from headend, device will be taxed for reassembly.
Hope it helps.
Thanks for your answers. Really appreciate it. I have question regarding VPN client and its issue with Intel Centrio Wireless. Hope that this still related with this event`s topic. Recently Intel announce that its Centrino chipset has flaws, see the following URL:
And, Nortel also announces that its Contivity client has problems with Centrino, see the following URL:
Does Cisco VPN client have any installation issues with Centrino based notebook also ? Any consideration to make a formal announce regarding compatibility of Centrino notebook and VPN3000 Client ??
to be able to counter flooding, we'd first need to see what type of flooding is that, once you find out the cause, you can try following:
Thanks for bringing it up.
Intel lists down us (cisco's vpn3000 client) as one of its verified vpn clients;
Hence, I'd doubt that we have any issue with this chipset (atleast so far), but just in case, if you have a laptop having issues with our vpn client, please feel free to open up a TAC case to track that issue with the relevant information (e.g., complete PC crash info., laptop hw details, sw versions involved etc.)
As far as I know, there are/were some issues with Intel Centrino and VPN-software.
These links provide some info:
These articles mainly discuss Nortel VPN software, but I also read in a Dutch article, that other vendors' VPN-software might have the same problems.
Hope this helps...
Since I'm new to the VPN world I hope my questions aren't too far off base.
I am curious about VPN to VPN connections. I have a vendor using VPN Client 3.X to access my 3005 Concentrator. I also have a PIX 515e firewall. He is able to get to my internal networks, but he is attempting to connect, via Terminal Services, to a computer on the DMZ of a remote site PIX. The tunnel to the remote site is also on that 3005.
My tech at the remote location solved this type problem for his own network by creating a separate tunnel on the PIX that specifically pointed to my DMZ, and the tunnel on the 3005 continued to point to my internal networks. It seems he could not access my DMZ through the PIX via the 3005, even though internal users can access the DMZ directly. Is this simply a routing issue? I am assuming the using the client to tunnel to the 3005 prevents you from using the 3005 to connect to the remote sites, since both tunnels are built on the external interface.
Any insight would be appreciated.
its a supported configuration, and should work fine.
Make sure the following:
1 - your network lists include relevant networks
2 - you are not running 3.6.7(A/B/C) versions, try using atleast 3.6.7D or later version of code (thats because of CSCea48892).
hope it helps.
Hi. We have created a Lan to Lan tunnel between a Symantec Enterprise Firewall (SEF) and Cisco VPN 3030 Concentrator. The network behind the SEF is a private 10.x.x.x net. The private side of the concentrator is a routable network. When we send traffic from behind the SEF to the destination Lan (not directly connected to the concentrator), the traffic comes out of the tunnel with a source of the private address even though we have configured a static NAT rule to translate the 10.x.x.x source. We have tried multiple configurations and have tested with two different Concentrators and SEF platforms with the same result...no NAT on lan to lan tunnels. How do we get L2L NAT to work in this scenario?
L2L NAT works for the traffic generated from inside the vpn3k(and same traffic when coming inbound), vpn3k wouldn't NAT these packets as in your Q, though a PIX/IOS can do that.
Private IP Addressing schemes on the private interfaces of the VPN GW is a common configuration, and perfectly acceptable as you use IPSec tunnel mode, which adds a tunnel header to the private IP payload (to be encrypted).
In your scenario, either you can use PIX/IOS as the ipsec endpoint, or NAT on the SEF side before encryption.
To understand the usage of L2L NAT on VPN3K, please point to :
Thanks very much!
if I read your response correctly, we can not NAT traffic through the concentrator that is sourced from the public interface on L2L tunnels. Unfortunately I can not perform the NAT at the SEF because the response to that traffic would not be routed back through the tunnel (address would be sent to default gateway not concentrator..static routes would work though but we frown on those). What about the case of overlapping subnets at the endpoints...isn't L2L NAT performed in that case even though the traffic comes through the public interface?
Hi. I am configuring LAN to LAN connections between two VPN 3030 boxes. The configurations will be IPSec based using SHA1 and 3DES. Is there a link at Cisco (or do you have) that details LAN to LAN configurations for vpn concentrators? Thanks
we dont seem to have a VPN3K to VPN3K Lan to Lan config, but you can follow this config to do that:
change transform-sets to SHA-1/3DES.
hope it helps.
Subject: External Interface usage of VPN30x0
We implemented to a customer a VPN30x0 that utilizes all of its available interfaces. The Public I/F (public checkbox is on) is for clients coming from the Internet, the External I/F (public checkbox is off) is for clients coming from the Internal LAN, the Private I/F (public checkbox is off) is for high security servers` segment located at the protected network internal LAN (this segment is virtually different location with the other internal LAN segments). The default route is to the Internet router. We created a filter for the External I/F so it can terminate IPSec connection. The rules are IKE, IPSec, UDP 10000 (for IPSec/UDP encapsulation). But we have problem that the VPN 30x0 hanging every 2~3 days and reboot the device only the way to solve the problem. Do you have any insight what might cause the problem ? The Concentrator is running with v3.6.7F.
Thanks in advance,
W/o taking a look at the concentrator Logs(IKE/IKEDBG/IPSec/IPSecDBG /w sev=1-10) it would be difficult to say anything, but I have these questions:
1 - what does "hang" mean, vpn3k doesn't respond to IP/vpn3k doesn't respond to vpn clients/vpn3k doesn't respond to console, please describe it?
2 - Do you get any crashdump, or any peculiar Log entries in the Logs (given that you have proper events turned on /w required severity) just before the vpn3k hangs?
3 - Does using the External Interface only for IPSec (no IPSec/UDP) clients help?
4 - What if you load this CONFIG (from this box) to another box, and try 4.0.1B code with the exact same CONFIGuration, does that help?
Let me know above details.
Subject : Tool to test throughput of VPN clients.
Do you have any idea what tool to use for testing the performance (throughput) of VPN 3000 clients (around 1000 users or more) ?
Appreciate for any pointer.
There is no single tool (ready made) that can do that, and in fact measuring the "Encrypted Throughput" wont give you any benefits, unless you measure cleartext Throughput for the same environment, and then compare the two to see how much performance is degraded(because of IPSec overhead).
VPN Client | Statistics, gives you details abt encrypted bytes sent/received, you can scale them on a "measured time period", and calculate encrypted throughput, but again there is no single tool that can do this manual process for you.
Can multiple iP addresses be assigned to a VPN 3005 concentrator's public interface?
Let's say that I have two ISP connections each with its own (albeit limited) IP address space; each ISP's address space has been mapped to the same "DMZ" subnet. The screening router for this subnet would have two IP addresses: a primary one from one ISP and a secondary one from the other ISP as shown below:
ip address a.b.c.1 255.255.255.240
ip address e.f.g.1 255.255.255.240 secondary
Is there a technique by which I could define the 3005's public interface as a.b.c.2/28 (primary) and e.f.g.2/28 (secondary)?
If not, could the screening router use NAT to readdress a.b.c.2 and e.f.g.2 to a private IP address?
If so, how would this be impacting on IPSEC, IPSEC/TCPand IPSEC/UDP traffic from VPN clients?
Thank you in advance for your reply.