traffic flow problem with tunnel created on network with an existing tunnel to a VPN concentrator

Answered Question
Dec 3rd, 2009

Hi, I have been working with Cisco and the vendor for 2 weeks on this.II am hoping the issue we are seeing will ring a bell with someone.

Some background information:

I am working a vendor who requires a site to site tunnel.  My network currently has 1 site to site with the vendor using a Juniper SSG which works without incident.  I am transitioning to Cisco 2811 routers and am set up a new tunnel with the vendor for the 2800 using another public ip in my address range.  So my network has 2 tunnels with the vendor who is using a Cisco VPN concentrator.  The hosts behind the tunnel use Public IP addresses 20x.x.x.x.

My Cisco router will create a tunnel, but I cannot get to the hosts on the vendor's network through the Cisco 2811, but I can through the Juniper tunnel.  The Vendor sees my packets and the vendors' host replies to them and sends them to tunnel.  They never reach the outside interface on my Cisco router.

I am NATing out of the outside interface so that my endpoint and peer are the same IP.  (note, I tried doing a static NAT and having a the tunnel address and my host address different to the same result.  Cisco confirmed that I do not require 2 different addresses and this setup has been successful with creating other successful tunnels toa  different network.)

I had tested this setup on a staging network before moving the router to the production network and my Cisco 2811 was able to create the tunnel and ping the inside host.  Once we moved the Router to the colo, we can no longer ping the host behind the vendor's tunnel.   The vendor has assured me that the tunnel setup is exactly the same and that he sees his host sending traffic to the tunnel.  The vendor does seem well versed with the VPN concentrator and successfully manages connections for many clients.

The vendor has a second VPN concentrator on a separate network and I can connect to that VPN concentrator successfully from the Cisco 2811 that is having issues with the Concentrator that also has a tunnel with the Juniper.

Here is what we have done so far:

1) Confirm the 2811 config with Cisco support.  The tunnel is up.  sh cyrpto ipa sa shows the tunnel up.
2) Enable Nat-T on the VPN Contrator side of the tunnel
3) Confirm that the trafiic flows properly from  a tunnel on another network (which would indicate the Cisco config is ok)
4) Successfully tunnel and reach hosts on a different config
5) Confirm all tunnel settings with the vendor
6) The vendor confirmed that the host on his side has no routes and points to the default gateway
7) Rebuild the tunnel from scratch
8) Confirm with our ISP that no routes are diverting the traffic else where.  My lSP gateway sees my outside address as directly connected. 
9) Confirm that the ACL matches with the vendor
10) I can not pull down the Juniper as it is in production and in constant use

Is there any known issue with using a VPN concentrator to connect to 2 tunnels on the same /28 network range? 

Any ideas or options are welcome.  I have had countless webex sessions with Cisco, but do not have access to the Vendor's concentrator.  I can pass along any suggestions. 

Here is some code


crypto isakmp policy 1
encr 3des
hash md5
authentication pre-share
group 2
!
crypto isakmp policy 2
encr 3des
authentication pre-share
group 2


crypto ipsec transform-set mytrans esp-aes esp-sha-hmac

crypto dynamic-map dynmap 30
set transform-set myset


crypto isakmp key <key here> address <vendor endpoint ip> no-xauth

interface FastEthernet0/0
description $ETH-LAN$$ETH-SW-LAUNCH$$INTF-INFO-FE 0/0$
ip address <my outside> 255.255.255.240
ip access-group 107 in
ip access-group 106 out
ip nat outside
ip virtual-reassembly
ip route-cache flow
duplex auto
speed auto
crypto map mymap

logging Access-lists (applied to outside to get a sense of what is coming in.  No esp traffic comes in, there are never hits)

access-list 106 permit esp host <my public IP>  host <vendor VPN endpoint> log
access-list 106 permit ip any any
access-list 107 permit esp host <vendor VPN host (is public IP)> host <my public IP> log
access-list 107 permit ip host <my public IP> host <vendor VPN host (is public IP)> log


access-list 107 permit ip host <my public IP>  host <vendor VPN endpoint> log
access-list 107 permit ip any any


sh crypto isa sa
IPv4 Crypto ISAKMP SA
dst             src             state          conn-id slot status
<vendor endpoint ip>  <my public ip>      QM_IDLE           1010    0 ACTIVE


Crypto Map "mymap" 1 ipsec-isakmp
        Peer = <vendor endpoint IP>
        Extended IP access list 116
            access-list 116 permit ip host <my outside ip> host <vendor inside out IP> (which is a public IP))
        Current peer: <vendor endpoint IP>
        Security association lifetime: 4608000 kilobytes/2800 seconds
        PFS (Y/N): N
        Transform sets={
                mytrans,
        }

I have this problem too.
0 votes

OK - so I messed around in the lab for 20 mins and came up with the below (ip's are test IP's:-


(4) ip nat pool crypto-nat 10.1.1.1 10.1.1.1 prefix-length 30 <> this is the New NAT Address

!
(1) ip nat inside source list 102 interface FastEthernet0/0 overload <> This is the default interface NAT

!
ip nat inside source route-map crypto-nat pool crypto-nat overload <> This is the policy based NAT

!

(6) access-list 101 permit ip 172.16.1.0 0.0.0.255 172.16.2.0 0.0.0.255 <> Defines the source and destination IP traffic

!

(2) access-list 102 deny   ip 172.16.1.0 0.0.0.255 172.16.2.0 0.0.0.255 <> Does not NAT the normal communication


(3) access-list 102 deny   ip host 10.1.1.1 172.16.2.0 0.0.0.255 <> Does not re-NAT the NAT


(1) access-list 102 permit ip 172.16.1.0 0.0.0.255 any <> allows all else to use the interface IP for NAT

!

(5) route-map crypto-nat permit 5 <> The condition for the specific required NAT
     match ip address 101 <> Match source and destination IP traffic required to be NAT'td

(7) access-list 103 permit ip host 10.1.1.1 172.16.2.0 0.0.0.255 <> crypto acl

So how the above works, when a packet with the source IP of 172.16.1.0/24 wants to leave the router to connect to say google, the source will change to the interface IP (1).  When 172.16.1.0/24 wants to speak to172.16.2.0/24, it does not get translated (2).  When the traffic the remote end has matched the next NAT clause - the already NAT'td IP will not be touched again(3) When a host 172.16.1.0/24 wants to communicate with 172.16.2.20/24 we have to NAT to a specific NAT pool is required (4).  We need to define a method of matching specific traffic to apply the NAT using a route map (5) which is only applied when specific traffic is in play (6) Then you just need to define the interesting traffic for the VPN to initiate and allow comms(7)

  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 5 (1 ratings)
Loading.
BIAAdvisor2009 Mon, 12/07/2009 - 08:31

Here are my ACLs

access-list 1 permit 10.200.0.0 0.0.255.255
access-list 100 permit ip 10.200.45.0 0.0.0.255 10.200.51.0 0.0.0.255
access-list 100 permit ip 10.200.44.0 0.0.0.255 10.200.51.0 0.0.0.255
access-list 100 permit ip 10.200.48.0 0.0.0.255 10.200.51.0 0.0.0.255
access-list 100 permit ip 10.200.49.0 0.0.0.255 10.200.51.0 0.0.0.255
access-list 100 permit ip 10.200.50.0 0.0.0.255 10.200.51.0 0.0.0.255
access-list 105 deny   ip 10.200.45.0 0.0.0.255 10.200.51.0 0.0.0.255
access-list 105 deny   ip 10.200.44.0 0.0.0.255 10.200.51.0 0.0.0.255
access-list 105 deny   ip 10.200.48.0 0.0.0.255 10.200.51.0 0.0.0.255
access-list 105 deny   ip 10.200.49.0 0.0.0.255 10.200.51.0 0.0.0.255
access-list 105 deny   ip 10.200.50.0 0.0.0.255 10.200.51.0 0.0.0.255
access-list 105 permit ip 10.200.0.0 0.0.255.255 any
access-list 115 permit ip host 72.x.x.x host 6x.xx.xx.x

ip nat inside source list 105 pool mypool overload

ACL 100 is applied here:

crypto isakmp client configuration group remotegroupname

acl 100

BIAAdvisor2009 Mon, 12/07/2009 - 09:07

Here it is.  Thank you very much for taking a look at this.

#show access-list 100
Extended IP access list 100
    10 permit ip 10.200.45.0 0.0.0.255 10.200.51.0 0.0.0.255
    20 permit ip 10.200.44.0 0.0.0.255 10.200.51.0 0.0.0.255
    30 permit ip 10.200.48.0 0.0.0.255 10.200.51.0 0.0.0.255
    40 permit ip 10.200.49.0 0.0.0.255 10.200.51.0 0.0.0.255
    50 permit ip 10.200.50.0 0.0.0.255 10.200.51.0 0.0.0.255
#show access-list 105
Extended IP access list 105
    10 deny ip 10.200.45.0 0.0.0.255 10.200.51.0 0.0.0.255
    20 deny ip 10.200.44.0 0.0.0.255 10.200.51.0 0.0.0.255
    30 deny ip 10.200.48.0 0.0.0.255 10.200.51.0 0.0.0.255
    40 deny ip 10.200.49.0 0.0.0.255 10.200.51.0 0.0.0.255
    50 deny ip 10.200.50.0 0.0.0.255 10.200.51.0 0.0.0.255 (3155 matches)
    60 permit ip 10.200.0.0 0.0.255.255 any (3 matches)

BIAAdvisor2009 Mon, 12/07/2009 - 09:15

FYI _ i have moved testing of the VPN to a secondary router.  Here is the config of (slightly different than above post - sorry about that.)

interface FastEthernet0/0
description $ETH-LAN$$ETH-SW-LAUNCH$$INTF-INFO-FE 0/0$
ip address 72.x.x.x 255.255.255.240
ip nat outside
ip virtual-reassembly
duplex auto
speed auto
crypto map mymap


crypto isakmp policy 1
encr 3des
hash md5
authentication pre-share
group 2
!
crypto isakmp policy 2
encr 3des
authentication pre-share
group 2
!
crypto isakmp policy 3
crypto isakmp key xxxxxxxxa address 1xx.x.x.51 no-xauth
!
crypto isakmp client configuration group remoteaccgroup
key xxxxxxx
pool ippool
acl 100
save-password
!
crypto isakmp peer address 1xx.x.x.51

crypto ipsec transform-set myset esp-3des esp-md5-hmac
crypto ipsec transform-set mytrans esp-aes esp-sha-hmac
!
crypto dynamic-map dynmap 30
set transform-set myset
!
!
crypto map mymap client authentication list userauthen
crypto map mymap isakmp authorization list groupauthor
crypto map mymap client configuration address respond
crypto map mymap 1 ipsec-isakmp
set peer 1xx.x.x.51
set security-association lifetime seconds 2800
set transform-set mytrans
match address 115
crypto map mymap 65000 ipsec-isakmp dynamic dynmap

ip local pool ippool 10.200.51.101 10.200.51.199
ip forward-protocol nd
ip route 0.0.0.0 0.0.0.0 72.x.x.x8

ip nat pool mypool 72.x.x.x 72.x.x.x prefix-length 28  (this is the same outside address as the WAN interface)
ip nat inside source list 105 pool mypool overload

OK - so apart from no traffic being seen on any of the below - which is a big worry

#show access-list 100

Extended IP access list 100

    10 permit ip 10.200.45.0 0.0.0.255 10.200.51.0 0.0.0.255

    20 permit ip 10.200.44.0 0.0.0.255 10.200.51.0 0.0.0.255

    30 permit ip 10.200.48.0 0.0.0.255 10.200.51.0 0.0.0.255

    40 permit ip 10.200.49.0 0.0.0.255 10.200.51.0 0.0.0.255

    50 permit ip 10.200.50.0 0.0.0.255 10.200.51.0 0.0.0.255

#show access-list 105

Extended IP access list 105

    10 deny ip 10.200.45.0 0.0.0.255 10.200.51.0 0.0.0.255

    20 deny ip 10.200.44.0 0.0.0.255 10.200.51.0 0.0.0.255

    30 deny ip 10.200.48.0 0.0.0.255 10.200.51.0 0.0.0.255

    40 deny ip 10.200.49.0 0.0.0.255 10.200.51.0 0.0.0.255

    50 deny ip 10.200.50.0 0.0.0.255 10.200.51.0 0.0.0.255 (3155 matches)

    60 permit ip 10.200.0.0 0.0.255.255 any (3 matches)

What networks should be working with the new VPN tunnel, and what are working with the existing tunnel???

BIAAdvisor2009 Mon, 12/07/2009 - 09:23

the existing working (juniper) and my cisco need to temporarily connect to teh same tunnel to access a vendor's FTP site.

The cisco is also acting as a VPN server for remote users to log into.  The back up cisco I am using for testing, doesnt get much use (it is the failover of an HSRP group).

The network that should be working with the VPN tunnel; is an endpoint with an address of

2xx.x.x.51 and a host behind it with a public address of 6x.x.x.1

The vendor uses public addresses behind its VPN concentrator.  NAT-T is now turned off on the vendor's concentrator. (as turning it on actually stopped teh tunnel from coming up at all.)

BIAAdvisor2009 Mon, 12/07/2009 - 19:43

FYI added the remote peer and endpoint to the no nat ACL - same issue

Extended IP access list 105
    5 deny ip host 7x.x.x.x81 host 6x.x.x.1 (10 matches)
    6 deny ip host 7x.x.x9.x81 host 2xx.x.x.51 (33 matches)
    10 deny ip 10.200.45.0 0.0.0.255 10.200.51.0 0.0.0.255
    20 deny ip 10.200.44.0 0.0.0.255 10.200.51.0 0.0.0.255
    30 deny ip 10.200.48.0 0.0.0.255 10.200.51.0 0.0.0.255
    40 deny ip 10.200.49.0 0.0.0.255 10.200.51.0 0.0.0.255
    50 deny ip 10.200.50.0 0.0.0.255 10.200.51.0 0.0.0.255

For your no-nat acl, you ONLY define what you do not want to be natt'd from source to destination, this does not include the external outside IP addresses, as these do need to be used for nat.

The issue you have is currently you have NO traffic that hits the below

Extended IP access list 105
    5 deny ip host 7x.x.x.x81 host 66.36.94.1 (10 matches)
    6 deny ip host 7x.x.x9.x81 host 2xx.x.x.51 (33 matches)
    10 deny ip 10.200.45.0 0.0.0.255 10.200.51.0 0.0.0.255
    20 deny ip 10.200.44.0 0.0.0.255 10.200.51.0 0.0.0.255
    30 deny ip 10.200.48.0 0.0.0.255 10.200.51.0 0.0.0.255
    40 deny ip 10.200.49.0 0.0.0.255 10.200.51.0 0.0.0.255
    50 deny ip 10.200.50.0 0.0.0.255 10.200.51.0 0.0.0.255

Where are the 10.200.45.x etc subnets? Are they on your LAN - are there actualy machines on those subnets?

BIAAdvisor2009 Tue, 12/08/2009 - 06:58

There are hosts on that network.  Now that I have switched to testing on the backup router, the hosts point top the primary.  I have been pinging from the router interface f0/1 (10.200.44.251).  I will point a host to the testing router and try from the host.  I have successlly pinging from the router for other site to site VPN tests with different networks.

I see what you mean about no hits.  The 105 ACL stops nat to the VPN client network (10.200.51.x).

Should I add an entry like

deny ip host 10.200.44.50 (host I will test with) host 6x.x.x.1

edit - I  just added this and pinged from the inside interface - it is getting hits.  Still no ICMP reply.

105 5 deny ip host 10.200.44.251 host 6x.x.x.1 (5 matches).

BIAAdvisor2009 Tue, 12/08/2009 - 07:19

one more point - the vendor requires we use a public address.  This being the case, wouldnt we want to NAT the traffic?  the vendor crypto ACL has the public address7x.x.x.x81 as the remote peer, not  10.200.x.x.

BIAAdvisor2009 Tue, 12/08/2009 - 07:29

Site a external IP 7x.x.x.x81

Site a LAN IP 10.200.0.0

Site b external IP 20x.x.x.51

Site b LAN IP 6x.x.x.1

Site B uses all public addresses.

Site B is set up to use my 7.x.x.x.81 as the LAN address.

so Site B ACL is permit host x.x.x.x81 host 6x.x.x.1

make sense?

Thanks so much for your help with this.

only if that IP address is on the internal network of the remote end, the below is an example vpn tunnel

Site a external IP 1.1.1.1

Site a LAN IP 192.168.1.0/24

Site b external IP 2.2.2.2

Site b LAN IP 172.16.1.0 24

My interesting traffic (enable the VPN tunnel) acl would read

ip access-list 101 permit ip 192.168.1.0 0.0.0.255 172.16.1.0 0.0.0.255

My no-nat acl would read

ip access-list 201 deny ip 192.168.1.0 0.0.0.255 172.16.1.0 0.0.0.255 do not nat before encryption

ip access-list 201 permit ip 192.168.1.0 0.0.0.255 any nat anything else that leaves me

So the above means

anything from 192.168.1.x to 172.16.1.0 encrypt

anything from 192.168.1.x to 172.16.1.0 DO NOT NAT

What are YOUR IP networks?

What are the REMOTE IP networks?

BIAAdvisor2009 Tue, 12/08/2009 - 07:42

FYI I posted my network info in the message above your last post.  I think that should cover it.  Hopefully that makes sense.

Correct Answer

OK - so I messed around in the lab for 20 mins and came up with the below (ip's are test IP's:-


(4) ip nat pool crypto-nat 10.1.1.1 10.1.1.1 prefix-length 30 <> this is the New NAT Address

!
(1) ip nat inside source list 102 interface FastEthernet0/0 overload <> This is the default interface NAT

!
ip nat inside source route-map crypto-nat pool crypto-nat overload <> This is the policy based NAT

!

(6) access-list 101 permit ip 172.16.1.0 0.0.0.255 172.16.2.0 0.0.0.255 <> Defines the source and destination IP traffic

!

(2) access-list 102 deny   ip 172.16.1.0 0.0.0.255 172.16.2.0 0.0.0.255 <> Does not NAT the normal communication


(3) access-list 102 deny   ip host 10.1.1.1 172.16.2.0 0.0.0.255 <> Does not re-NAT the NAT


(1) access-list 102 permit ip 172.16.1.0 0.0.0.255 any <> allows all else to use the interface IP for NAT

!

(5) route-map crypto-nat permit 5 <> The condition for the specific required NAT
     match ip address 101 <> Match source and destination IP traffic required to be NAT'td

(7) access-list 103 permit ip host 10.1.1.1 172.16.2.0 0.0.0.255 <> crypto acl

So how the above works, when a packet with the source IP of 172.16.1.0/24 wants to leave the router to connect to say google, the source will change to the interface IP (1).  When 172.16.1.0/24 wants to speak to172.16.2.0/24, it does not get translated (2).  When the traffic the remote end has matched the next NAT clause - the already NAT'td IP will not be touched again(3) When a host 172.16.1.0/24 wants to communicate with 172.16.2.20/24 we have to NAT to a specific NAT pool is required (4).  We need to define a method of matching specific traffic to apply the NAT using a route map (5) which is only applied when specific traffic is in play (6) Then you just need to define the interesting traffic for the VPN to initiate and allow comms(7)

BIAAdvisor2009 Fri, 12/11/2009 - 07:01

Thanks so much for your help.  I just tested this and it works as you explained.  It looks like the other side had some type of configuration or hardware problem, so I am now working.  Your posted config is very educational.  Much appreciated!

Actions

This Discussion