I have a Cisco 1921 router, which I can connect to from the internet using the software VPN Client for Windows.
I can ping the inside interface of the router when connected.
However, I cannot ping any devices on the inside subnet.
Here is the config for the router:
Update 1: Diagram
Is it a problem that I'm using "10.70.12.x" for the VPN Client pool, and the entire rest of the "10.70.x.x/16" range is the local subnet? In other words, is this VPN Client/local devices "overlap" a potential routing problem?
I believe it has something to do with access lists.
I was had a similar issue the other day and the fix was to explicitly permit the traffic.
However, my understanding of one of the differences between Cisco 1921's and ASA's is that ASA's "deny unless explicitly permitted" and 1921's "permit unless explicitly denied."
And I don't see any deny that should effect this particular VPN client.
I went over the configuration and found that you are missing the NAT exemption rule for the VPN clients. We will need to configure a deny rule on access-list 130 in order to bypass the global NAT when the packets are coming back from the inside network (10.70.0.0/16) to the ip pool (10.70.12.0/24). Please make sure that you enter the line before the permit rule.
ip access-l ext 130
1 deny ip 10.70.0.0 0.0.255.255 10.70.12.0 0.0.0.255
Hope this helps,
Tried your suggestion:
PG-1921(config-ext-nacl)#do show access-list 130 Extended IP access list 130 10 deny ip 10.70.0.0 0.0.255.255 10.20.0.0 0.0.255.255 (42952 matches) 20 deny ip 10.70.0.0 0.0.255.255 18.104.22.168 0.255.255.255 (1122 matches) 30 deny ip 10.70.0.0 0.0.255.255 10.80.0.0 0.0.255.255 (10440 matches) 40 deny ip 10.70.0.0 0.0.255.255 10.40.0.0 0.0.255.255 (21780 matches) 50 permit ip 10.70.0.0 0.0.255.255 any (7280 matches) PG-1921(config-ext-nacl)#1 deny ip 10.70.0.0 0.0.255.255 10.70.12.0 0.0.0.255 PG-1921(config-ext-nacl)#do show access-list 130 Extended IP access list 130 1 deny ip 10.70.0.0 0.0.255.255 10.70.12.0 0.0.0.255 (6 matches) 10 deny ip 10.70.0.0 0.0.255.255 10.20.0.0 0.0.255.255 (42952 matches) 20 deny ip 10.70.0.0 0.0.255.255 22.214.171.124 0.255.255.255 (1122 matches) 30 deny ip 10.70.0.0 0.0.255.255 10.80.0.0 0.0.255.255 (10456 matches) 40 deny ip 10.70.0.0 0.0.255.255 10.40.0.0 0.0.255.255 (21844 matches) 50 permit ip 10.70.0.0 0.0.255.255 any (7343 matches)
Unfortunately, the problem seems to persist.
At least we could see hitcounts on the ACL rule. That means that the packets are trying to get back to the VPN client. Could you please check if there is any encrypted packet on the Security Association? You could use the show crypto ipsec sa peer x.x.x.x (public ip address of the VPN client), in order to check that. Also, if there is any other device on the core network taking care of the routing please make sure that there is a route for the 10.70.12.0/24 network to send the packets back to the router (VPN endpoint).
Hope this help you,
I suspect we are not actually seeing the hitcounts on the ACL rule.
Pings to 10.70.1.1 (the router inside interface) continue to succeed, and increment the ACL hitcounter.
However, pings to 10.70.2.10 (a host on inside subnet) still fail, and do not affect the ACL hitcounter at all.
Here is the result of show crypto ipsec sa peer
As you could see clearly from the outputs there are a lot of decaps packets and few encaps. What that means is that the packets sent from the VPN client to the inside network are being routed properly, however, the packets coming back from the inside network to the VPN client are not reaching this router, that's why we see only few encaps from the ping test to the inside interface.
As I mentioned before we will need to configure the proper routing on the inside network for the packets destine to the ip pool range used for the VPN clients (10.70.12.0/24). One way would be using static routing. I do not know how many hops there are between the router and the host that you are trying to reach from the VPN client, but we will need to make sure that we have the routes configured for the packets coming back to the clients. Once that's properly configured the router will receive the packets, encrypt them and send them back through the Tunnel.
#pkts encaps: 6, #pkts encrypt: 6, #pkts digest: 6
#pkts decaps: 902, #pkts decrypt: 902, #pkts verify: 902
I hope this help you with your issue,
When VPN Clients connect, it appears that routes are automatically added, so what need is there to add a route?
Output of show ip route:
Looks like VPN Clients do have routes out to the next hop... Don't they?
You're right, every time that a VPN client connects the router will add a temporal route to reach the assigned ip address through the interface where the VPN client connection was established. However, as I explained you the issue is not with the routes on this router, the problem is with the routing on the inside network. Remember that the packets are not even reaching the router when coming back from the inside to the VPN client. That's why we need to make sure that the proper routes are configured for the 10.70.12.0/24 network on the core to return the packets back through the VPN endpoint (router).
For instance let's say that I have the following environment:
SERVER----------in--CORE ROUTER out-----in- Router (VPN endpoint)-out--ISP
(192.168.1.10) (192.168.1.1) (172.16.1.1) (172.16.1.2) (126.96.36.199)
Lets' say that my VPN connects to the ip address 188.8.131.52. The router will configure a route automatically for the VPN client pointing to the ISP:
S 10.70.12.5/32 [1/0] via 184.108.40.206, GigabitEthernet0/0
When the VPN client tries to reach the 192.168.1.10 server the packet will reach the router then the router will send the paket to the core router and finally the packet will reach the Server.
10.70.12.5 =====> 172.16.1.1====>192.168.1.10
When the packet is coming back the server does not know how to reach the 10.70.12.5 ip address , so it will send the packet to the defaut gateway 192.168.1.1 (core inside interface). The Core router will receive the packet sourced from the Server to the VPN client, but it does not how to reach the 10.70.12.5 either, so it will send the packet through the default gateway. The tricky part is if the gateway of the core is not the 172.16.1.2 the router will not ever receive that packet.
192.168.1.10====> 192.168.1.1====> ?????
So we will need to add a route on the core router to let him know how to route the packets when going to the 10.70.12.0/24 destination.
ip route 10.70.12.0 255.255.255.0 172.16.1.2
Hope this clear you doubts,
I understand what you're saying about how a core router can impede "traffic trying to return to VPN clients".
However, is it possible it's something else? Because I'm being told by the client that, no, there is only one router (the one for which I've shared the config), and that there are no other routing devices on the inside network, just switches.
Understand, but the question is why the packets are not reaching the router back? It's also important to check the basics as well. Please make sure that you are not running any powerful Anti-Virus/Firewall system on the local host that could be blocking this traffic. You could also get captures on the end host using Wireshark and see if the packets coming from the VPN client are really reaching the device. Next step will be to make sure that there is not any firewall on between, and finally get span sessions on the switches to see if the packets are moving back and forth through the correct ports. Also, check any security settings on the switches such as VLAN access-map.
Hope this helps,
I gained access to one of the hosts local to the router.
The Windows firewall is disabled.
It appears that pings from VPN Client users reach the host OK.
And it looks like ping reponses are going back out.
Is it possible that the traffic is in fact making its way to the router, but being dropped because of some bad NAT logic? I have a vague hunch that maybe NAT exemption is related, but not sure...
Here's the NAT exemption rule:
access-list 130 deny ip 10.70.2.0 0.0.0.255 10.70.12.0 0.0.0.255
access-list 130 deny ip 10.70.2.0 0.0.0.255 10.99.99.0 0.0.0.255
access-list 130 deny ip 10.70.0.0 0.0.255.255 10.20.0.0 0.0.255.255
access-list 130 deny ip 10.70.0.0 0.0.255.255 220.127.116.11 0.255.255.255
access-list 130 deny ip 10.70.0.0 0.0.255.255 10.80.0.0 0.0.255.255
access-list 130 deny ip 10.70.0.0 0.0.255.255 10.40.0.0 0.0.255.255
access-list 130 deny ip 10.70.0.0 0.0.255.255 10.45.0.0 0.0.255.255
access-list 130 permit ip 10.70.0.0 0.0.255.255 any
So if I'm pinging 10.70.2.10 from 10.99.99.3, it should be exempt from NAT. Hmm. So I'd say that's correct...
However, I've noticed that if I do ipconfig on the VPN Client machine, the subnet mask is 255.0.0.0. Could this non /24 subnet mask be an issue?
Thanks for the information!
Based on the capture we could see that the VPN client is getting the ip address 10.70.4.17 which is not part of the range specified on the ip local pool (10.70.12.1-50), this is definitely related to the subnet mask. I also noticed that you are trying to reach the 10.99.99.3 host, which means that we need a NAT exemption rule for the packets coming back from 10.99.99.0/24 to 10.70.12.0/24.
Please specify the subnet mask that you would like to use under the crypto isakmp client configuration. Below you will find the outputs:
crytpo isakmp client configuration group VPNCLIENTGROUP
Also, please add the NAT exemption rule for the 10.99.99.0/24 network.
host(config)#ip access-list ext 130
host(config-ext-nacl)# 5 deny 10.99.99.0 0.0.0.255 10.70.12.0 0.0.0.255
Then reconnect the VPN client and check the mask, that should be a /24 mask. If that's correct please try to reach the 10.99.99.3 host again.
I hope this helps,
I'm sorry, I changed the VPN Client pool to 10.99.99.1-50 because I've been told it's better practice to have the pool be outside of the inside-subnet (and I just wanted to simplify things while troubleshooting).
So actually, I'm trying to ping 10.70.4.17 from the VPN Client (10.99.99.3).
In which case, the NAT exemption should actually look like this, right?
access-list 130 deny ip 10.70.0.0 0.0.255.255 10.99.99.0 0.0.0.255
And it does.
I've also made sure to set the VPN Client pool mask already. The VPN Client's receive a subnet mask of /24.
Same result. Everything I've said in this post was already true when I wrote the previous one. Any other ideas?
Thanks for the information!
I did not know that you changed the ip local pool. In that case you just need to make sure that the NAT exemption rule is included before the permit ip 10.70.0.0 0.0.255.255 any. On the ACLs attached on the comment before you did not include the access-list 130 deny ip 10.70.0.0 0.0.255.255 10.99.99.0 0.0.0.255. The only ACL rule referring to the 10.99.99.0/24 network was access-list 130 deny ip 10.70.2.0 0.0.0.255 10.99.99.0 0.0.0.255, which is only allowing the 10.70.2.0/24 network to communicate with the VPN clients. Please make sure that the ACL is added like the one on the previous comment, and as I said before the permit.
host(config)#ip access-list ext 130
host(config-ext-nacl)# 1 deny ip 10.70.0.0 0.0.255.255 10.99.99.0 0.0.0.255
After that we should be good on the router site. If the problem continues please check any device on between and make sure it is forwarding the packets back to the router. You could also place an access-group on the inside interface of the router for incoming packets, that will help you to see if the packets are really coming back from the server.
host(config)#ip access-list ext 111
host(config-ext-nacl)#10 permit ip host 10.70.4.17 10.99.99.0 0.0.0.255 log
host(config-ext-nacl)#20 permit ip any any
host(config-if)#ip access-group 111 in
Then, if the packets are really coming back to the Router you should see hit counts on the ACL rule with the command show access-list 111.
I hope this helps,
I added the ACL like you suggested, however the issue persisted. I ended up giving Cisco support a call, and this is what ended up being the final bit that got it working. Although I'm not entirely sure why.
Thanks for all your help. I really appreciate it!