Welcome to the Cisco Networking Professionals Ask the Expert conversation. This is an opportunity to discuss Troubleshooting IPsec on VPN 3000 Concentrators with Cisco expert Jazib Frahim. Jazib belongs to the VPN-Solutions Team in Research Triangle Park, North Carolina where he acts as the Team Lead for his theatre. He joined Cisco Systems in 1999 as an engineer in the Technical Assistance Center (TAC). Feel free to post any questions relating to Troubleshooting IPsec VPN 3000 Concentrators. Remember to use the rating system to let Jazib know if youve received an adequate response.
Jazib might not be able to answer each question due to the volume expected during this event. Our moderators will post many of the unanswered questions in other discussion forums shortly after the event. This event lasts through December 13. Visit this forum often to view responses to your questions and the questions of other community members.
Hi, Just a following up with ya. Looking for documentation on errors(event class) from monitoring for the 3000. I thought with 3.6 it was gonna be on CCO after a couple of weeks? Any new status or can you send a link?
Nice to hear back from you. I had a discussion with the Engineering Development group and according to them, it is still under works. I'll follow up with the DE group again on it and let you know
Just got an answer from the development team. It seems like they are shooting for early next year
Hope it helps
I am also interested in a common error explanation reference. One we're dealing with lately is "session reset by peer" messages seen at the client PC. What are common reasons for seeing this?
If you are seeing "session reset by peer", then it means the concentrator disconnected your session for some reason.
1) If the client hit the Max-Connect timeout
2) If the client hit the idle timeout
3) if the session is administratively disconnected by the concentrator
4) If the concentrator does not receive the keepalives
are a few important ones
Concentrator logs will give a better idea why it happened
Hope that helps
Is there a way to make the keep-alive process more forgiving to poor environments? For example say you have some folks coming to your concentrator from India which is renowed for its congested pipes out of the country...might you be able to accomodate those users coming from congested or high latency environments by adjusting a parameter on the CVPN?
For the users who have slow Internet connections, what you can do is, create a new new group and disable "IKE Keepalive" under the IPSec tab. This will allow the users to be connected even if IKE keepalives are getting lost
Hopw that helps
say - one question related to that... If you have a pix 506E connected to a CVPN concentrator - is there anything you need to do on the PIX side so it does not do IKE keepalives?
I have 100 vpn clients connect to my concentrator 3000 around the world ,some of the user from certain area complained that they couldn't make the ipsec tunnel up with the error message "peer response timeout". How to fix it?
It seems like either the client didn't get a proper response from the concentrator on time that it timed out the connection. If you enable "log viewer" on the client, and set the filter to High for the different classes, you will see what happening. If your client is timing out the connection, you will see a lot of retransmission in the logs
Hope that gives you an idea of what's going on
You got no options here on the concentrator. If you tried setting the peer to the HSRP address already, then this probably isn't going to work. As you already have found out using the same access-list as interesting traffic on the concentrator with 2 separate L2L connections wont work. There isn't much you can do on the router side either. If you replace the 3000 with a router, then you can do multiple set peer x.x.x.x with one crypto map which would work for your scenario.
Kurtis, I am a little puzzeled by setting a local or non public address as a peer address. Is that what you meen by setting the peer to the HSRP address? The HSRP address we use is a private address because we do not have a public address that could be shared by the seperate circuits. I have also looked at some things on the Cisco site but am confused how they talk about using loopback addresses for local addresses for peer addresses. See below:
If you use a local-address for that crypto map set, it has multiple effects:
Only one IPSec security association database will be established and shared for traffic through both interfaces.
The IP address of the specified interface will be used as the local address for IPSec (and IKE) traffic originating from or destined to that interface.
One suggestion is to use a loopback interface as the referenced local address interface, because the loopback interface never goes down.
I am not sure how the above would work? Could the 3005 see a loopback interface?
I assumed you was running hsrp on the outside as well. My mistake. Using the loopback is also assuming that you have one router with 2 pipes and not 2 routers with a separate pipe on each. I dont think that this is possible with your set up. In order for this to work you will need to have a single ip address for termination which is where the loopback comes into place. Even if you run HSRP on the outside interfaces I dont think you can use the ip address for a loopback on both routers. Maybe if you set up the loopbacks with HSRP. Im not able to test that out for ya, maybe you could give the results. Didn't realize I was posting in Jazibs forum, maybe he will add some further insight.
Your right, unfortunately HRSP is only running on the inside. Our goal is automated IPSec failover and we have the capability to put the two pipes on the same router and have the other router on site but off the network for hardware redundancy. We have one location now with the two pipes in one router setup and it also has the same negative result .
I am not sure how the remote router loopback works with the 3005? If I enter a crypto map local address loopback 0 and give loopback 0 a private IP (again .. the two pipes have unique public addresses) on the router how will the 3005 associate the SA with the remote router if the peer address is private? Am I understanding this process correctly?
The loopback will have to be a public address in order for that to work as it wont work with a private. 3005 will work to the router in only that condition. So your understanding is correct that it wont work that way. Personally, I think you aren't using the right equipment for what your trying to do. You would be better off doing GRE over IPSEC with router to router. This will satisfy the load balancing you are trying to do on your routers. Just an idea.
As Kurtis mentioned, there is not a good way to get what you are looking for.
The only good option that I could think of, is to have a Cisco router ( rather than a concentrator) at the enterprise site.
By doing that, you can 2 options:
1) Either configure GRE over IPSec between your Enterprise router and multiple spoke routers. The good thing about this option is, all the routing updates will be sent between the sites dynamically and thus your enterprise router can failover to the other router fairly quickly.
2) or configure just native IPSec between your Enterprise router and your spoke routers, and specify multiple set peer statements at the enterprise site for the backup connection.
Sorry there is no good method to achieve redundancy if you mix IOS routers and VPN concentrators
Hope that helps
Thanks Jazib, that gives me something to think about. Having another device (IOS router or PIX) to terminate the backup sessions at the Enterprise site may be what I am looking for because it will add another element of redundancy which is key.
I think using an IOS router would be a better choice than a pix firewall. If you want to go for GRE over IPSec model, pix does not have this functionality.
With the PIS FW, you can, however, use multiple set peer statements for redundancy
Hope that helps
3005 Concentrator at enterprise location - and cookie cutter template for remote sites that use IOS 1720 and 2610 IOS routers for IPSec LAN2LAN connections to the enterprise location. Each remote location uses two IOS routers each using a unique ISP T1 circuit and public address but each use HSRP to share a common Ethernet interface as a gateway for the local network traffic.
On the 3005 I have to configure primary and backup IPSec LAN2LAN sessions for each location because each of the two remote IOS routers at each location cannot share a common public IP for the peer address and the Concentrator (to my knowledge) will not allow more than one peer address without setting up the session as though the peer address was dynamic.
Because I am using the two tunnels for redundancy, they share the same address list information for encrypting traffic for the tunnel. On the IOS router side I have tried using a lower priority number for isakmp policy and crypto map statements that match the second or backup LAN2LAN 3005 tunnel and use different lifetime settings so that if the first LAN2LAN session is available it will negotiate the primary LAN2LAN session if the peer responds. IKE keepalives are enabled on both ends.
My issue is that because the two LAN2LAN tunnels to the same site share the same local network address range there does not appear to be a way to control how the 3005 chooses which remote peer to contact to initiate a session via the enterprise side of the WAN. If primary peer at address 208.x.x.x does not respond for LAN addresses 10.56.x.x the 3005 does not try the other secondary session which also encrypts traffic to network 10.56.x.x for LAN traffic to peer 63.x.x.x. The remote side however is able to bring up the backup or secondary LAN2LAN session from their side but because we cannot initiate things under these circumstances from the enterprise side this scenario does not fully meet our needs.
Is it possible to setup a lan to lan tunnel between two private network which have same ip subnet? If possible, please tell me how can I configure the vpn equipments. Thanks.
I believe these are some of the web-link that you might be interested in
Hope that helps
I have a question with regards to running ipsec utilizing Cisco IOS. I have been working on a Cisoc IOS vpn network that is running ipsec with gre tunnels and nat between sites. Split tunneling has been enabled by using the nat to translate private traffic before it is routed to the internet. Intranet (VPN) traffic is of course encrypted and routed via the gre tunnels to the remote sites. I was recently analyzing the nat tables and came accross entries with both private source and destination addresses. RFC 1918 addresses are being used throughout the network for internal addressing. This tells me that vpn traffic is also being nated before it is encrypted. This should not be the case, public internet traffic should be the only traffic that is nated. VPN traffic should not be nated but rather encapsulated with a gre header to make it routable on the internet to the other endpoint of the tunnel. I'm wondering if this is the correct behavior? With this particular setup, is it neccessary to apply the appropriate deny access-list to the nat access-list to stop private traffic from be nated. I was under the impression that a deny nat statement should only be needed if running tunnel mode ipsec and nat only. My logic assumed that the gre interface would break the nat process because the ip nat inside and outside are applied only the ethernet and serial interfaces respectivally and not the tunnel interface. If tunnel mode ipsec and nat only were being used, that seems logical to apply the appropriate deny access-list statement to ensure only private (intranet) traffic is encrypted and not nated because all of the traffic is crossing the same interfaces.
any assistance is appreciated,
If the traffic is destined over the GRE tunnel and you don't have "ip nat inside/outside" command on the GRE tunnel itself, then your traffic should not be natted. In "sh ip nat trans", it should show you inside local, inside global and outside global. If the traffic is destined to something on the other side of the tunnel, then you will see remote private destination as outside global.
If you like, you can send me the output to me at: email@example.com
I'm having a problem getting 3.5.3 clients updating to 3.6.2.b.
I go to configuration-system-client update-entries
When clients connect, and click on the "launch" button they get a "this page cannot be displayed" message.
I must mention that I created my own customized executable using installshield and modified it to have our company logo as the backgroud, as well as 4 default connections included. The file name now is vpnclient.exe. I have that file inside a share called vpn on a internal server.
Is there any special configuration that has to be done on the server where the updated client is?
On the URL box of the 3000, I tried different paths, and changed permissions on the vpn share etc, but nothing seems to work,
I've been gradually setting up quite a number of cisco3000 concentrators with LAN-LAN connections, but i am having major trouble getting the routing to happen properly.
It seems that when i create LAN - LAN connections between any two sites, both sides can route properly and i can verify this by looking at the routing table. I'm using Network AutoDiscovery on all routers.
If i try to route across two concentrators this doesn't work:
SiteA ------------- SiteB ----------------------- SiteC
10.199.1.0 10.1.50.0 10.199.2.0
255.255.255.0 255.255.248.0 255.255.255.0
site a can see site b, site b can see both, and site c can see site b.
It seems to me that RIP entries do not get passed across multiple concentrators.
i've played with the rip entries on both the private and public interfaces in the hope that this will fix the problem.
I've also tried putting in reverse route injection and specifying the the local and remote subnets as well and this seems to get the routes across to the various places but still doesn't work.
i might add that the above diagram is a simplified version of what we have. we've got between 25-40 3000 VPNs around the place.
i've also been trying to put something useful in the Network Lists(Configuration | Policy Management | Traffic Management | Network Lists) in the hope that this will also fix our problem, but i get the error:
Error adding entries to network list (Instance Error).
even if i use the Generate Local List button.
hmm, i hope this is clear enough...
thanks and regards
Oh, couple more things i forgot to mention:
1. how does one troubleshoot incorrect routing?
a. doing a traceroute from a machine gets all * for every Cisco that it passes though, and gets * for dropouts as well. this is very confusing.
b. routing table on a cisco gets a lot of RIP entries, but it's quite hard to tell where they've come from except if you delete a LAN2LAN connection and watch them dissapear (after a clear routes).
2. one LAN2LAN connection i've made to a VPN in france refuses to route. i delete the LAN to LAN and the RIP entries on either end dissapear and then i recreate it, it connects etc etc to Phase 2 (in the event log) but refuses to route.
3. I've noticed on some connections (Administration | Administer Sessions)there is more than one IPSec Session with different subnets in them. one connection i've got with a static route the remote end (siteC) out to a standard router, gets two entries. the siteC rnage and also the siteD range (siteD is 10.199.8.0). so i would have thought that siteA would have both siteB,C,D in it's IPSec entries:
SiteA ------------- SiteB ----------------------- SiteC --------------------------SiteD (not VPN)
10.199.1.0------ 10.1.50.0------------------- 10.199.2.0 ---------------- 10.199.8.0
255.255.255.0 -255.255.248.0 -------------255.255.255.0 -----------255.255.255.0
but it doesn't....
anyway, sorry this last question is vague...
1a) for most part, concentrator show all the the next-hop ip addresses past the IPSec tunnel. Does your traceroute die at the private interface of the concentrator.?
1b) in the routing table, it should show you which concentrator is the next hop ip address for a specific subnet. You don't have to clear the Lan-Lan tunnel
2) when you are sending the packets over the tunnel, do you see any encrypts going up for that tunnel
3) The IPSec SAs are only creates when there is some interesting traffic to bring up the SA. If you don't see an SA, then there is a possibility that your concentrator never saw any interesting traffic to bring up the IPSec SA
Hope that helps
1a) yes it does, it just drops the actual cisco which is fine with the other things to watch.
1b) ah, this is most interesting, my routing table is nothing of the kind:
(edited) 203.x.x.1 is the gateway on the public address.
0.0.0.0 0.0.0.0 203.x.x.1 2 Default 0 1
10.1.0.0 255.255.248.0 203.x.x.1 2 RIP 23 2
10.1.16.0 255.255.248.0 203.x.x.1 2 RIP 23 9
10.1.24.0 255.255.248.0 203.x.x.1 2 RIP 23 9
10.1.40.0 255.255.248.0 203.x.x.1 2 RIP 23 9
10.1.64.0 255.255.248.0 0.0.0.0 1 Local 0 1
10.1.88.0 255.255.248.0 203.x.x.1 2 RIP 4 2
10.1.120.0 255.255.248.0 203.x.x.1 2 RIP 16 2
10.1.128.0 255.255.248.0 203.x.x.1 2 RIP 23 9
i've been playing with the rip settings on all these devices and our middle VPN (siteb in the ex) i've got RIP v1/v2 for both inbound and outbound on both the public and private ip.
i think the default is just inbound on the private which baffles me, if all the vpns have RIP only inbound how does the RIP ever get out to get to other VPNs?
what should they be for site A, Site B, and Site C? all set to the default? or site B something different because it should be passing stuff though?
2) good, that's something to watch for.
3) ah, so an IKE up means the link is up, that's happening. but for some reason no interesting traffic is getting through. i've got two VPNs with this behaviour, one works when i've got rip on both the public and private interface, and the other i'm struggling with cos i can't get anything though. :)