Cisco Support Community
cancel
Showing results for 
Search instead for 
Did you mean: 
New Member

VPN return traffic not flowing over the tunnel

Hello.

I've tried to find something on the internet to solve this, but am failing miserably. I guess, I really don't understand how the cisco decides on routing.

Anyway, I have a Cisco 837 which I'm using for internet access and which I would like to be able to terminate a VPN on. When I vpn in (using vpnc from a Solaris box as it happens which is connected to the ethernet interface of the cisco), I can establish an VPN and when I ping a host on the inside, I can see that ping packet arrive, however the return packet, the cisco 837 tries to send over the public internet facing interface Dialer1 without encryption. I can't for the life of me work out why.

(Also note: I can also establish a tunnel from the public internet, but again, I can't get any traffic back through the tunnel. I assume that I'm having the same issue, ie return packets aren't going where they should, but I don't know that for certain, on the host being pinged though, I can see the ping packets arriving and the host responding with an ICMP Echo reply).

here is the cisco version:

adsl#show version
Cisco IOS Software, C850 Software (C850-ADVSECURITYK9-M), Version 12.4(15)T5, RELEASE SOFTWARE (fc4)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2008 by Cisco Systems, Inc.
Compiled Thu 01-May-08 02:07 by prod_rel_team

ROM: System Bootstrap, Version 12.3(8r)YI4, RELEASE SOFTWARE

adsl uptime is 1 day, 19 hours, 27 minutes
System returned to ROM by power-on
System restarted at 17:20:56 bst Sun Oct 10 2010
System image file is "flash:c850-advsecurityk9-mz.124-15.T5.bin"

Cisco 857 (MPC8272) processor (revision 0x300) with 59392K/6144K bytes of memory.
Processor board ID FCZ122391F5
MPC8272 CPU Rev: Part Number 0xC, Mask Number 0x10
4 FastEthernet interfaces
1 ATM interface
128K bytes of non-volatile configuration memory.
20480K bytes of processor board System flash (Intel Strataflash)

Configuration register is 0x2102

And here is the cisco's configuration (IP address, etc changed of course):

Current configuration : 7782 bytes
!
! Last configuration change at 11:57:21 bst Mon Oct 11 2010 by bautsche
! NVRAM config last updated at 11:57:22 bst Mon Oct 11 2010 by bautsche
!
version 12.4
no service pad
service tcp-keepalives-in
service tcp-keepalives-out
service timestamps debug datetime msec localtime show-timezone
service timestamps log datetime msec localtime show-timezone
service password-encryption
service sequence-numbers
!
hostname adsl
!
boot-start-marker
boot-end-marker
!
logging buffered 4096
enable secret 5 <secret>
!
aaa new-model
!
!
aaa authentication login local_authen local
aaa authentication login sdm_vpn_xauth_ml_1 local
aaa authorization exec local_author local
aaa authorization network sdm_vpn_group_ml_1 local
!
!
aaa session-id common
clock timezone gmt 0
clock summer-time bst recurring last Sun Mar 1:00 last Sun Oct 1:00
!
!
dot11 syslog
no ip source-route
ip dhcp database dhcpinternal
no ip dhcp use vrf connected
ip dhcp excluded-address 10.10.7.1 10.10.7.99
ip dhcp excluded-address 10.10.7.151 10.10.7.255
!
ip dhcp pool dhcpinternal
   import all
   network 10.10.7.0 255.255.255.0
   default-router 10.10.7.1
   dns-server 212.159.6.9 212.159.6.10 212.159.13.49 212.159.13.50
!
!
ip cef
ip auth-proxy max-nodata-conns 3
ip admission max-nodata-conns 3
no ip bootp server
ip host nfs1 10.10.140.207
ip name-server 212.159.11.150
ip name-server 212.159.13.150
!
!
!
username cable password 7 <password>
username bautsche password 7 <password>
username vpnuser password 7 <password>
!
!
crypto isakmp policy 1
encr 3des
hash md5
authentication pre-share
group 2
!
crypto isakmp policy 2
encr aes 256
authentication pre-share
group 2
!
crypto isakmp policy 3
encr 3des
authentication pre-share group 2
crypto isakmp client configuration address-pool local SDM_POOL_1
!
crypto isakmp client configuration group groupname2
key <key>
dns 10.10.140.201 10.10.140.202
domain swangage.co.uk
pool SDM_POOL_1
max-users 3
netmask 255.255.255.0
!
crypto isakmp client configuration group groupname1
key <key>
dns 10.10.140.201 10.10.140.202
domain swangage.co.uk
pool SDM_POOL_1
max-users 3
netmask 255.255.255.0
crypto isakmp profile sdm-ike-profile-1
   match identity group groupname2
   client authentication list sdm_vpn_xauth_ml_1
   isakmp authorization list sdm_vpn_group_ml_1
   client configuration address respond
crypto isakmp profile sdm-ike-profile-2
   match identity group groupname1
   isakmp authorization list sdm_vpn_group_ml_1
   client configuration address respond
!
!
crypto ipsec transform-set ESP-3DES-SHA esp-3des esp-sha-hmac
crypto ipsec transform-set ESP_MD5_3DES esp-3des esp-md5-hmac
crypto ipsec transform-set ESP-AES-256-SHA esp-aes esp-sha-hmac
!
crypto dynamic-map SDM_DYNMAP_1 1
set security-association idle-time 3600
set transform-set ESP-AES-256-SHA
reverse-route
crypto dynamic-map SDM_DYNMAP_1 2
set security-association idle-time 3600
set transform-set ESP-AES-256-SHA
reverse-route
!
!
crypto map SDM_CMAP_1 client authentication list sdm_vpn_xauth_ml_1
crypto map SDM_CMAP_1 isakmp authorization list sdm_vpn_group_ml_1
crypto map SDM_CMAP_1 65535 ipsec-isakmp dynamic SDM_DYNMAP_1
!
crypto ctcp port 10000
archive
log config
  hidekeys
!
!
ip tcp synwait-time 10
!
!
!
interface Null0
no ip unreachables
!
interface ATM0
no ip address
no ip redirects
no ip unreachables
no ip proxy-arp
ip route-cache flow
no atm ilmi-keepalive
pvc 0/38
  encapsulation aal5mux ppp dialer
  dialer pool-member 1
!
dsl operating-mode auto
hold-queue 224 in
!
interface FastEthernet0
!
interface FastEthernet1
!
interface FastEthernet2
!
interface FastEthernet3
!
interface Vlan1
description $FW_INSIDE$
ip address 10.10.7.1 255.255.255.0
ip access-group 121 in
no ip redirects
no ip unreachables
no ip proxy-arp
ip nat inside
ip virtual-reassembly
ip route-cache flow
crypto map SDM_CMAP_1
hold-queue 100 out
!
interface Dialer1
description $FW_OUTSIDE$
ip address negotiated
ip access-group 121 in
no ip redirects
no ip unreachables
no ip proxy-arp
ip nat outside
ip virtual-reassembly
encapsulation ppp
ip route-cache flow
no ip split-horizon
dialer pool 1
dialer idle-timeout 0
dialer persistent
dialer-group 1
no cdp enable
ppp authentication chap callin
ppp chap hostname <hostname>
ppp chap password 7 <password>
crypto map SDM_CMAP_1
!
ip local pool SDM_POOL_1 10.10.148.11 10.10.148.20
ip local pool public_184 123.12.12.184
ip local pool public_186 123.12.12.186
ip local pool public_187 123.12.12.187
ip local pool internal_9 10.10.7.9
ip local pool internal_8 10.10.7.8
ip local pool internal_223 10.10.7.223
ip local pool internal_47 10.10.7.47
ip forward-protocol nd
ip route 0.0.0.0 0.0.0.0 Dialer1
ip route 10.10.140.0 255.255.255.0 10.10.7.2
!
no ip http server
no ip http secure-server
ip nat inside source route-map SDM_RMAP_1 interface Dialer1 overload
ip nat inside source static 10.10.7.9 123.12.12.184
ip nat inside source static tcp 10.10.7.8 22 123.12.12.185 22 extendable
ip nat inside source static tcp 10.10.7.8 25 123.12.12.185 25 extendable
ip nat inside source static tcp 10.10.7.8 80 123.12.12.185 80 extendable
ip nat inside source static tcp 10.10.7.8 443 123.12.12.185 443 extendable
ip nat inside source static tcp 10.10.7.8 993 123.12.12.185 993 extendable
ip nat inside source static tcp 10.10.7.8 1587 123.12.12.185 1587 extendable
ip nat inside source static tcp 10.10.7.8 8443 123.12.12.185 8443 extendable
ip nat inside source static 10.10.7.223 123.12.12.186
ip nat inside source static 10.10.7.47 123.12.12.187
!
logging 10.10.140.213
access-list 18 permit any
access-list 23 permit 10.10.140.0 0.0.0.255
access-list 23 permit 10.10.7.0 0.0.0.255
access-list 100 remark SDM_ACL Category=2
access-list 100 deny   ip any 10.10.148.0 0.0.0.255
access-list 100 permit ip any any
access-list 121 remark SDM_ACL Category=17
access-list 121 deny   udp any eq netbios-dgm any
access-list 121 deny   udp any eq netbios-ns any
access-list 121 deny   udp any eq netbios-ss any
access-list 121 deny   tcp any eq 137 any
access-list 121 deny   tcp any eq 138 any
access-list 121 deny   tcp any eq 139 any
access-list 121 permit ip any any
access-list 125 permit tcp any any eq www
access-list 125 permit udp any eq isakmp any
access-list 125 permit udp any any eq isakmp
access-list 194 deny   udp any eq isakmp any
access-list 194 deny   udp any any eq isakmp
access-list 194 permit ip host 123.12.12.184 any
access-list 194 permit ip any host 123.12.12.184
access-list 194 permit ip host 10.10.7.9 any
access-list 194 permit ip any host 10.10.7.9
access-list 195 deny   udp any eq isakmp any
access-list 195 deny   udp any any eq isakmp
access-list 195 permit ip host 123.12.12.185 any
access-list 195 permit ip any host 123.12.12.185
access-list 195 permit ip host 10.10.7.8 any
access-list 195 permit ip any host 10.10.7.8
no cdp run
route-map public_185 permit 10
match ip address 195
!
route-map public_184 permit 10
match ip address 194
!
route-map SDM_RMAP_1 permit 1
match ip address 100
!
!
control-plane
!
!
line con 0
login authentication local_authen
no modem enable
transport preferred none
transport output telnet
stopbits 1
line aux 0
login authentication local_authen
transport output telnet
stopbits 1
line vty 0 4
access-class 23 in
privilege level 15
authorization exec local_author
login authentication local_authen
length 0
transport preferred none
transport input telnet ssh
!
scheduler max-task-time 5000
scheduler allocate 4000 1000
scheduler interval 500
sntp server 130.88.202.49
sntp server 130.88.200.98
sntp server 130.88.200.6
sntp server 130.88.203.64
end

Any help would be appreciated.

Thanks a lot.

Ciao,

Eric

Everyone's tags (4)
1 ACCEPTED SOLUTION

Accepted Solutions
Cisco Employee

Re: VPN return traffic not flowing over the tunnel

Hi Eric,

(sorry for the delayed response - was in need of some vacation )

So I see you got a few steps farther now. I think there are 2 things we can try:

1)

I suppose you have put back this:

ip nat inside source route-map SDM_RMAP_1 interface Dialer1 overload

Since the routemap refers to ACL 100 to define the traffic to be translated, we can exclude the traffic that initiates from the router:

access-list 100 remark SDM_ACL Category=2

access-list 100 deny ip host 123.12.12.185 any
access-list 100 deny   ip any 10.10.148.0 0.0.0.255
access-list 100 permit ip any any

That should prevent the udp source port from changing from 4500 to 1029

OR

2)

if you prefer to use another ip address for VPN,

then you can use a loopback like this:

interface loopback 0

  ip address 123.12.12.187 255.255.255.255

  no shut

crypto map SDM_CMAP_1 local-address loopback 0

I don't think you need to apply the crypto map to the loopback interface, but it's been a while since I configured something like this, so if you have any trouble try that first, and if still not working get the crypto debugs again (isakmp+ipsec on the vpn router, nat+packet on the client router).

hth

Herbert

40 REPLIES
Cisco Employee

Re: VPN return traffic not flowing over the tunnel

Hey Eric,

When connected to the VPN, please post the output of "show crypto isa sa" and "show crypto ipsec sa". Also, what ip addresses are you trying to connect to when connected over VPN and how exactly are you trying to connect to them? Are you able to ping the router's IP address of 10.10.7.1 form the VPN clients?

Thanks and Regards,

Prapanch

New Member

Re: VPN return traffic not flowing over the tunnel

Hi Prapanch.

Thanks for your help.

Here are the outputs from show crypto isa sa:

adsl#show crypto isa sa
IPv4 Crypto ISAKMP SA
dst             src             state          conn-id slot status
123.12.12.185   10.10.7.115   QM_IDLE           2030    0 ACTIVE

IPv6 Crypto ISAKMP SA

adsl#

And the output from show crypto ipsec sa:

adsl#show crypto ipsec sa

interface: Dialer1
    Crypto map tag: SDM_CMAP_1, local addr 123.12.12.185

   protected vrf: (none)
   local  ident (addr/mask/prot/port): (0.0.0.0/0.0.0.0/0/0)
   remote ident (addr/mask/prot/port): (10.10.148.14/255.255.255.255/0/0)
   current_peer 10.10.7.115 port 500
     PERMIT, flags={}
    #pkts encaps: 0, #pkts encrypt: 0, #pkts digest: 0
    #pkts decaps: 21, #pkts decrypt: 21, #pkts verify: 21
    #pkts compressed: 0, #pkts decompressed: 0
    #pkts not compressed: 0, #pkts compr. failed: 0
    #pkts not decompressed: 0, #pkts decompress failed: 0
    #send errors 0, #recv errors 8

     local crypto endpt.: 123.12.12.185, remote crypto endpt.: 10.10.7.115
     path mtu 1500, ip mtu 1500, ip mtu idb Dialer1
     current outbound spi: 0xFF2842BF(4280828607)

     inbound esp sas:
      spi: 0x7C9990A0(2090438816)
        transform: esp-aes esp-sha-hmac ,
        in use settings ={Tunnel, }
        conn id: 285, flow_id: Motorola SEC 1.0:285, crypto map: SDM_CMAP_1
        sa timing: remaining key lifetime (k/sec): (4514693/3440)
        IV size: 16 bytes
        replay detection support: Y
        Status: ACTIVE

     inbound ah sas:

     inbound pcp sas:

     outbound esp sas:
      spi: 0xFF2842BF(4280828607)
        transform: esp-aes esp-sha-hmac ,
        in use settings ={Tunnel, }
        conn id: 286, flow_id: Motorola SEC 1.0:286, crypto map: SDM_CMAP_1
        sa timing: remaining key lifetime (k/sec): (4514698/3440)
        IV size: 16 bytes
        replay detection support: Y
        Status: ACTIVE

     outbound ah sas:

     outbound pcp sas:
adsl#

I'm trying to connect to 10.10.140.93 by using ping. The ping (Echo request) packet arrives at that node and the node responds with an echo reply but that is then transmitted in the clear out of Vlan1 (when the VPN client is on the 10.10.7.0/24 network, I can't see what happens when the client is on the public internet side).

Re pinging 10.10.7.1, that address pings when the client is on the 10.10.7.0/24 network (obviously) whether VPN'ed in or not, but it does not ping when VPN'ed in from the public internet.

Thanks again for your help.

Eric

Cisco Employee

Re: VPN return traffic not flowing over the tunnel

Hi ,

I see that there is a crypto map applied on the interface VLAN 1. Are you trying to terminate any tunnel on that interface ? If no please take out that command and try once more.

thanks,

Namit

New Member

Re: VPN return traffic not flowing over the tunnel

I've removed the crypto map on Vlan1 but that hasn't changed the behaviour, unfortunately. ICMP pings still arrive at the host being pinged, but the replys never make it to the VPN client. :-(

Eric

Cisco Employee

Re: VPN return traffic not flowing over the tunnel

Eric,

I see you have 2 groups, one with and one  without authentication - is that intentional? If not, remove one of the  groups ance everything associated with it.

If it is your intention  to have the 2 groups side by side, then you'll want to reference your  isakmp profiles in your crypto map:

crypto dynamic-map SDM_DYNMAP_1 1
    set isakmp profile sdm-ike-profile-1

crypto dynamic-map SDM_DYNMAP_1 2

   set isakmp profile sdm-ike-profile-2

Still, this does not explain that you see the packets going out on the inside interface without encryption...

What is the source and destination IP address and MAC address of those reply packets?

Then check:

show ip route

Assuming you hace CEF enabled:

show ip cef detail

And apart from all that, you may want to upgrade to a more recent IOS version like 12.4(15)T14 - I'm not saying this will solve this problem but it wouldn't hurt to exclude the possibility you're running into a known bug.

hth


Herbert

New Member

Re: VPN return traffic not flowing over the tunnel

Hallo Herbert.

You are quite correct about useless stuff hanging around in the configuration, there were a few left overs from testing that I did, I have now cleaned up my configuration but unfortunately, it's still now working.

Here's the configuration as it is now and then I'll put the output from the show ip commands.

Configuration:

Current configuration : 7323 bytes
!
! Last configuration change at 07:58:13 bst Thu Oct 14 2010 by bautsche
! NVRAM config last updated at 07:58:14 bst Thu Oct 14 2010 by bautsche
!
version 12.4
no service pad
service tcp-keepalives-in
service tcp-keepalives-out
service timestamps debug datetime msec localtime show-timezone
service timestamps log datetime msec localtime show-timezone
service password-encryption
service sequence-numbers
!
hostname adsl
!
boot-start-marker
boot-end-marker
!
logging buffered 4096
enable secret 5
!
aaa new-model
!
!
aaa authentication login local_authen local
aaa authentication login sdm_vpn_xauth_ml_1 local
aaa authorization exec local_author local
aaa authorization network sdm_vpn_group_ml_1 local
!
!
aaa session-id common
clock timezone gmt 0
clock summer-time bst recurring last Sun Mar 1:00 last Sun Oct 1:00
!
!
dot11 syslog
no ip source-route
ip dhcp database dhcpinternal
no ip dhcp use vrf connected
ip dhcp excluded-address 10.10.7.1 10.10.7.99
ip dhcp excluded-address 10.10.7.151 10.10.7.255
!
ip dhcp pool dhcpinternal
   import all
   network 10.10.7.0 255.255.255.0
   default-router 10.10.7.1
   dns-server 212.159.6.9 212.159.6.10 212.159.13.49 212.159.13.50
!
!
ip cef
ip auth-proxy max-nodata-conns 3
ip admission max-nodata-conns 3
no ip bootp server
ip host nfs1 10.10.140.207
ip name-server 212.159.11.150
ip name-server 212.159.13.150
!
!
!
username cable password 7 091E4A210051
username bautsche password 7 0832495C1F4D0812
username vpnuser password 7 04764A051D2E7B161B1C
!
!
crypto isakmp policy 1
encr 3des
hash md5
authentication pre-share
group 2
crypto isakmp client configuration address-pool local SDM_POOL_1
!
crypto isakmp client configuration group
key
dns 10.10.140.201 10.10.140.202
domain
pool SDM_POOL_1
max-users 3
netmask 255.255.255.0
crypto isakmp profile sdm-ike-profile-1
   match identity group
   client authentication list sdm_vpn_xauth_ml_1
   isakmp authorization list sdm_vpn_group_ml_1
   client configuration address respond
!
!
crypto ipsec transform-set ESP-3DES-SHA esp-3des esp-sha-hmac
crypto ipsec transform-set ESP_MD5_3DES esp-3des esp-md5-hmac
crypto ipsec transform-set ESP-AES-256-SHA esp-aes esp-sha-hmac
!
crypto dynamic-map SDM_DYNMAP_1 1
set security-association idle-time 3600
set transform-set ESP-AES-256-SHA
reverse-route
!
!
crypto map SDM_CMAP_1 client authentication list sdm_vpn_xauth_ml_1
crypto map SDM_CMAP_1 isakmp authorization list sdm_vpn_group_ml_1
crypto map SDM_CMAP_1 65535 ipsec-isakmp dynamic SDM_DYNMAP_1
!
crypto ctcp port 10000
archive
log config
  hidekeys
!
!
ip tcp synwait-time 10
!
!
!
interface Null0
no ip unreachables
!
interface ATM0
no ip address
no ip redirects
no ip unreachables
no ip proxy-arp
ip route-cache flow
no atm ilmi-keepalive
pvc 0/38
  encapsulation aal5mux ppp dialer
  dialer pool-member 1
!
dsl operating-mode auto
hold-queue 224 in
!
interface FastEthernet0
!
interface FastEthernet1
!
interface FastEthernet2
!
interface FastEthernet3
!
interface Vlan1
description $FW_INSIDE$
ip address 10.10.7.1 255.255.255.0
ip access-group 121 in
no ip redirects
no ip unreachables
no ip proxy-arp
ip nat inside
ip virtual-reassembly
ip route-cache flow
hold-queue 100 out
!
interface Dialer1
description $FW_OUTSIDE$
ip address negotiated
ip access-group 121 in
no ip redirects
no ip unreachables
no ip proxy-arp
ip nat outside
ip virtual-reassembly
encapsulation ppp
ip route-cache flow
no ip split-horizon
dialer pool 1
dialer idle-timeout 0
dialer persistent
dialer-group 1
no cdp enable
ppp authentication chap callin
ppp chap hostname
ppp chap password 7
crypto map SDM_CMAP_1
!
ip local pool SDM_POOL_1 10.10.148.11 10.10.148.20
ip local pool public_184 123.10.10.184
ip local pool public_186 123.10.10.186
ip local pool public_187 123.10.10.187
ip local pool internal_9 10.10.7.9
ip local pool internal_8 10.10.7.8
ip local pool internal_223 10.10.7.223
ip local pool internal_47 10.10.7.47
ip forward-protocol nd
ip route 0.0.0.0 0.0.0.0 Dialer1
ip route 10.10.140.0 255.255.255.0 10.10.7.2
!
no ip http server
no ip http secure-server
ip nat inside source route-map SDM_RMAP_1 interface Dialer1 overload
ip nat inside source static 10.10.7.9 123.10.10.184
ip nat inside source static tcp 10.10.7.8 22 123.10.10.185 22 extendable
ip nat inside source static tcp 10.10.7.8 25 123.10.10.185 25 extendable
ip nat inside source static tcp 10.10.7.8 80 123.10.10.185 80 extendable
ip nat inside source static tcp 10.10.7.8 443 123.10.10.185 443 extendable
ip nat inside source static tcp 10.10.7.8 993 123.10.10.185 993 extendable
ip nat inside source static tcp 10.10.7.8 1587 123.10.10.185 1587 extendable
ip nat inside source static tcp 10.10.7.8 8443 123.10.10.185 8443 extendable
ip nat inside source static 10.10.7.223 123.10.10.186
ip nat inside source static 10.10.7.47 123.10.10.187
!
logging 10.10.140.213
access-list 18 permit any
access-list 23 permit 10.10.140.0 0.0.0.255
access-list 23 permit 10.10.7.0 0.0.0.255
access-list 100 remark SDM_ACL Category=2
access-list 100 deny   ip any 10.10.148.0 0.0.0.255
access-list 100 permit ip any any
access-list 101 permit ip 10.10.148.0 0.0.0.255 any
access-list 101 permit ip any 10.10.148.0 0.0.0.255
access-list 101 deny   ip any any
access-list 121 remark SDM_ACL Category=17
access-list 121 deny   udp any eq netbios-dgm any
access-list 121 deny   udp any eq netbios-ns any
access-list 121 deny   udp any eq netbios-ss any
access-list 121 deny   tcp any eq 137 any
access-list 121 deny   tcp any eq 138 any
access-list 121 deny   tcp any eq 139 any
access-list 121 permit ip any any
access-list 125 permit tcp any any eq www
access-list 125 permit udp any eq isakmp any
access-list 125 permit udp any any eq isakmp
access-list 194 deny   udp any eq isakmp any
access-list 194 deny   udp any any eq isakmp
access-list 194 permit ip host 123.10.10.184 any
access-list 194 permit ip any host 123.10.10.184
access-list 194 permit ip host 10.10.7.9 any
access-list 194 permit ip any host 10.10.7.9
access-list 195 deny   udp any eq isakmp any
access-list 195 deny   udp any any eq isakmp
access-list 195 permit ip host 123.10.10.185 any
access-list 195 permit ip any host 123.10.10.185
access-list 195 permit ip host 10.10.7.8 any
access-list 195 permit ip any host 10.10.7.8
no cdp run
route-map public_185 permit 10
match ip address 195
!
route-map public_184 permit 10
match ip address 194
!
route-map SDM_RMAP_1 permit 1
match ip address 100
!
!
control-plane
!
!
line con 0
login authentication local_authen
no modem enable
transport preferred none
transport output telnet
stopbits 1
line aux 0
login authentication local_authen
transport output telnet
stopbits 1
line vty 0 4
access-class 23 in
privilege level 15
authorization exec local_author
login authentication local_authen
length 0
transport preferred none
transport input telnet ssh
!
scheduler max-task-time 5000
scheduler allocate 4000 1000
scheduler interval 500
sntp server 130.88.202.49
sntp server 130.88.200.98
sntp server 130.88.200.6
sntp server 130.88.203.64
end

And here the output from the show route commands (ip addresses change again, 122.1.1.142 is the IP of the client on the public internet):

adsl#show ip route 10.10.148.12
Routing entry for 10.10.148.12/32
  Known via "static", distance 1, metric 0
  Routing Descriptor Blocks:
  * 122.1.1.142
      Route metric is 0, traffic share count is 1

adsl#show ip route 10.10.140.93
Routing entry for 10.10.140.0/24
  Known via "static", distance 1, metric 0
  Routing Descriptor Blocks:
  * 10.10.7.2
      Route metric is 0, traffic share count is 1

adsl#sh ip cef 10.10.148.12 detail
10.10.148.12/32, version 27, epoch 0, cached adjacency to Dialer1
0 packets, 0 bytes
  via 122.1.1.142, 0 dependencies, recursive
    next hop 122.1.1.142, Dialer1 via 0.0.0.0/0
    valid cached adjacency
adsl#show ip cef 10.10.140.93 detail
10.10.140.0/24, version 11, epoch 0, cached adjacency 10.10.7.2
0 packets, 0 bytes
  via 10.10.7.2, 0 dependencies, recursive
    next hop 10.10.7.2, Vlan1 via 10.10.7.2/32
    valid cached adjacency
adsl#

I can still see the pings arrive and being responded to on the target system 10.10.140.93, by the way.

Thanks again for everyone's help, it's really appreciated.

Eric

Cisco Employee

Re: VPN return traffic not flowing over the tunnel

Eric,

could you still tie the ike profile to the crypto map :

crypto dynamic-map SDM_DYNMAP_1 1
    set isakmp profile sdm-ike-profile-1

If that does not help, can you confirm that

- the echo reply packets arrive at the router, and are sent back out the vlan1 interface? If so what is the source and destination IP address and MAC address of the reply packets you see going out of vlan1

- "show crypto ipsec sa" shows no packets are being encrypted

Herbert

New Member

Re: VPN return traffic not flowing over the tunnel

Hello Herbert.

I have added set iskmp sdm-ike-profile-1 to the SDM_DYNMAP_1 1.

It's not fixed the issue, so re your other questions:

>  can you confirm that the echo reply packets arrive at the router, and are sent back out the vlan1 interface?

I can't really as I don't know how to make the router tell me. What I can tell you is that my firewall passes them back to the router:

This is a snoop of the interface of the firewall connected to vlan 10.10.7.0/24:

[...]

vpn-adsl-01 -> oberon ICMP Echo request (ID: 949 Sequence number: 49)
oberon -> vpn-adsl-01 ICMP Echo reply (ID: 949 Sequence number: 49)

[...]

And this is the firewall's routing table (extract):

root@tethys # netstat -rn

Routing Table: IPv4
  Destination           Gateway           Flags  Ref     Use     Interface
-------------------- -------------------- ----- ----- ---------- ---------
[....]

10.10.7.0          10.10.7.2          U         1          0 xnf1   
[....]

> If so what is the source and destination IP address and MAC address of the reply packets you see going out of vlan1

I exect that the reply packets go out of Dialer1 rather than vlan1 given the routing configuration on the router:

adsl#sh ip route
Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route

Gateway of last resort is 0.0.0.0 to network 0.0.0.0

     123.0.0.0/32 is subnetted, 1 subnets
C       123.1.1.185 is directly connected, Dialer1
     10.10.148.0/32 is subnetted, 1 subnets
S       10.10.148.11 [1/0] via 83.217.167.142
     180.16.128.0/32 is subnetted, 1 subnets
C       180.16.128.227 is directly connected, Dialer1
S    10.10.140.0/24 [1/0] via 10.10.7.2
C    10.10.7.0/24 is directly connected, Vlan1
S*   0.0.0.0/0 is directly connected, Dialer1

> "show crypto ipsec sa" shows no packets are being encrypted

adsl#show crypto ipsec sa

interface: Dialer1
    Crypto map tag: SDM_CMAP_1, local addr 84.92.202.185

   protected vrf: (none)
   local  ident (addr/mask/prot/port): (0.0.0.0/0.0.0.0/0/0)
   remote ident (addr/mask/prot/port): (10.10.148.11/255.255.255.255/0/0)
   current_peer port 500
     PERMIT, flags={}
    #pkts encaps: 529, #pkts encrypt: 529, #pkts digest: 529
    #pkts decaps: 529, #pkts decrypt: 529, #pkts verify: 529
    #pkts compressed: 0, #pkts decompressed: 0
    #pkts not compressed: 0, #pkts compr. failed: 0
    #pkts not decompressed: 0, #pkts decompress failed: 0
    #send errors 0, #recv errors 0

     local crypto endpt.: 123.1.1.185, remote crypto endpt.:
     path mtu 1500, ip mtu 1500, ip mtu idb Dialer1
     current outbound spi: 0x125571BF(307589567)

     inbound esp sas:
      spi: 0x49714BA8(1232161704)
        transform: esp-aes esp-sha-hmac ,
        in use settings ={Tunnel, }
        conn id: 1, flow_id: Motorola SEC 1.0:1, crypto map: SDM_CMAP_1
        sa timing: remaining key lifetime (k/sec): (4501781/3216)
        IV size: 16 bytes
        replay detection support: Y
        Status: ACTIVE

     inbound ah sas:

     inbound pcp sas:

     outbound esp sas:
      spi: 0x125571BF(307589567)
        transform: esp-aes esp-sha-hmac ,
        in use settings ={Tunnel, }
        conn id: 2, flow_id: Motorola SEC 1.0:2, crypto map: SDM_CMAP_1
        sa timing: remaining key lifetime (k/sec): (4501770/3214)
        IV size: 16 bytes
        replay detection support: Y
        Status: ACTIVE

     outbound ah sas:

     outbound pcp sas:
adsl#

I'm afraid I'm not clear on how to read the output of that command though...

Sorry.

Eric

Cisco Employee

Re: VPN return traffic not flowing over the tunnel

Hi Eric

well I think the problem with the external client may be different than the one with the client on the inside.

In the latest output of the external client we see:

    #pkts encaps: 529, #pkts encrypt: 529, #pkts digest: 529

    #pkts decaps: 529, #pkts decrypt: 529, #pkts verify: 529

I.e. it is encrypting as many packets as it is decrypting - which seems to indicate that the echo replies get encrypted ok.

So can you please check your client, does it have similar counters for packets sent/received (encrypted/decrypted)?

Or could you sniff the traffic on the client to see if it is receiving IPsec (ESP) packets?

Is the external client behind a NAT device?

Could it be that the ISP to which the VPNrouter is connected, does not allow outbound IPsec packets?

Herbert

New Member

Re: VPN return traffic not flowing over the tunnel

Thanks for your help.

You have made me check again something I had checked earlier, ie. the snoop on the client. Earlier, the client simply didn't get any packets back, but now it seems it does but doesn't appear know what to do with them (so we - you really - have fixed something, which is encouraging):

10.10.40.104 -> 123.1.1.185 ESP SPI=0x824af60f Replay=56
123.1.1.185 -> 10.10.40.104 ESP SPI=0x583b2299 Replay=55
10.10.40.104 -> 123.1.1.185 ICMP Destination unreachable (Bad protocol 50)
10.10.40.104 -> 123.1.1.185 ESP SPI=0x824af60f Replay=57
123.1.1.185 -> 10.10.40.104 ESP SPI=0x583b2299 Replay=56
10.10.40.104 -> 123.1.1.185 ICMP Destination unreachable (Bad protocol 50)
10.10.40.104 -> 123.1.1.185 ESP SPI=0x824af60f Replay=58
10.10.40.104 -> 123.1.1.185 ESP SPI=0x824af60f Replay=59
123.1.1.185 -> 10.10.40.104 ESP SPI=0x583b2299 Replay=57
10.10.40.104 -> 123.1.1.185 ICMP Destination unreachable (Bad protocol 50)
123.1.1.185 -> 10.10.40.104 ESP SPI=0x583b2299 Replay=58
10.10.40.104 -> 123.1.1.185 ICMP Destination unreachable (Bad protocol 50)

The ICMP Destination unreachable packets basically complain about the ESP packet that just came in with "Bad protocol".

The client is behind a NAT device (actually a Cisco 871 ADSL router). It's not likely that the ISP is having an issue with IPsec though as I can VPN into my employer's VPN service using the same tools just fine.

Thanks again for your persistence in helping me to debug this.

Eric

Cisco Employee

Re: VPN return traffic not flowing over the tunnel

Ok, glad to see we're making progress

Now I asked about the ISP and the NAT device because I thought  they might be blackholing the return packets, but your snoop confirms that they effectively arrive on the client, so this is now definitely a client issue.

Googling the unreachable message, I found this:

http://thestaticvoid.com/post/2010/07/22/fun-with-vpnc/

So could you please try to run vpnc with  --natt-mode force-natt

hth

Herbert

New Member

Re: VPN return traffic not flowing over the tunnel

Hi Herbert.

Thanks again for your help.

I was using cisco-udp as the NAT mode. I have changed it to force-natt but that unfortunately doesn't establish the tunnel. Here is the debug output from the Cisco router when trying with force-natt (IP addresses altered again). vpnc exits with "no response from target" after waiting about 30secs:

000631: Oct 14 12:49:35.282 bst: ISAKMP:(2006):purging node 427214974
000632: Oct 14 12:49:35.282 bst: ISAKMP:(2006):purging node 2060102796
000633: Oct 14 12:49:35.282 bst: ISAKMP:(2006):purging node -1191592158
000634: Oct 14 12:49:35.282 bst: ISAKMP:(2006):purging node 228
000635: Oct 14 12:49:35.282 bst: ISAKMP:(2006):purging node 195
000636: Oct 14 12:49:37.465 bst: ISAKMP (0:0): received packet from 120.3.3.142 dport 500 sport 500 Global (N) NEW SA
000637: Oct 14 12:49:37.465 bst: ISAKMP: Created a peer struct for 120.3.3.142, peer port 500
000638: Oct 14 12:49:37.465 bst: ISAKMP: New peer created peer = 0x82F6A850 peer_handle = 0x80000009
000639: Oct 14 12:49:37.465 bst: ISAKMP: Locking peer struct 0x82F6A850, refcount 1 for crypto_isakmp_process_block
000640: Oct 14 12:49:37.465 bst: ISAKMP:(0):Setting client config settings 82C3B344
000641: Oct 14 12:49:37.465 bst: ISAKMP:(0):(Re)Setting client xauth list  and state
000642: Oct 14 12:49:37.465 bst: ISAKMP/xauth: initializing AAA request
000643: Oct 14 12:49:37.465 bst: ISAKMP: local port 500, remote port 500
000644: Oct 14 12:49:37.469 bst: ISAKMP: Find a dup sa in the avl tree during calling isadb_insert sa = 8210A588
000645: Oct 14 12:49:37.469 bst: ISAKMP:(0): processing SA payload. message ID = 0
000646: Oct 14 12:49:37.469 bst: ISAKMP:(0): processing ID payload. message ID = 0
000647: Oct 14 12:49:37.469 bst: ISAKMP (0:0): ID payload
        next-payload : 13
        type         : 11
        group id     : groupname
        protocol     : 17
        port         : 500
        length       : 18
000648: Oct 14 12:49:37.469 bst: ISAKMP:(0):: peer matches sdm-ike-profile-1 profile
000649: Oct 14 12:49:37.469 bst: ISAKMP:(0):(Re)Setting client xauth list sdm_vpn_xauth_ml_1 and state
000650: Oct 14 12:49:37.469 bst: ISAKMP/xauth: initializing AAA request
000651: Oct 14 12:49:37.469 bst: ISAKMP:(0): processing vendor id payload
000652: Oct 14 12:49:37.469 bst: ISAKMP:(0): vendor ID seems Unity/DPD but major 242 mismatch
000653: Oct 14 12:49:37.469 bst: ISAKMP:(0): vendor ID is XAUTH
000654: Oct 14 12:49:37.469 bst: ISAKMP:(0): processing vendor id payload
000655: Oct 14 12:49:37.469 bst: ISAKMP:(0): vendor ID is Unity
000656: Oct 14 12:49:37.469 bst: ISAKMP:(0): processing vendor id payload
000657: Oct 14 12:49:37.469 bst: ISAKMP:(0): vendor ID seems Unity/DPD but major 69 mismatch
000658: Oct 14 12:49:37.469 bst: ISAKMP (0:0): vendor ID is NAT-T RFC 3947
000659: Oct 14 12:49:37.469 bst: ISAKMP:(0): processing vendor id payload
000660: Oct 14 12:49:37.469 bst: ISAKMP:(0): vendor ID seems Unity/DPD but major 123 mismatch
000661: Oct 14 12:49:37.469 bst: ISAKMP:(0): vendor ID is NAT-T v2
000662: Oct 14 12:49:37.469 bst: ISAKMP:(0): processing vendor id payload
000663: Oct 14 12:49:37.469 bst: ISAKMP:(0): vendor ID seems Unity/DPD but major 164 mismatch
000664: Oct 14 12:49:37.469 bst: ISAKMP:(0): processing vendor id payload
000665: Oct 14 12:49:37.469 bst: ISAKMP:(0): vendor ID seems Unity/DPD but major 168 mismatch
000666: Oct 14 12:49:37.473 bst: ISAKMP:(0): processing vendor id payload
000667: Oct 14 12:49:37.473 bst: ISAKMP:(0): vendor ID seems Unity/DPD but major 221 mismatch
000668: Oct 14 12:49:37.473 bst: ISAKMP:(0): processing vendor id payload
000669: Oct 14 12:49:37.473 bst: ISAKMP:(0): vendor ID is DPD
000670: Oct 14 12:49:37.473 bst: ISAKMP:(0): Authentication by xauth preshared
000671: Oct 14 12:49:37.473 bst: ISAKMP:(0):Checking ISAKMP transform 0 against priority 1 policy
000672: Oct 14 12:49:37.473 bst: ISAKMP:      keylength of 256
000673: Oct 14 12:49:37.473 bst: ISAKMP:      encryption AES-CBC
000674: Oct 14 12:49:37.473 bst: ISAKMP:      hash SHA
000675: Oct 14 12:49:37.473 bst: ISAKMP:      default group 2
000676: Oct 14 12:49:37.473 bst: ISAKMP:      auth XAUTHInitPreShared
000677: Oct 14 12:49:37.473 bst: ISAKMP:      life type in seconds
000678: Oct 14 12:49:37.473 bst: ISAKMP:      life duration (VPI) of  0x0 0x20 0xC4 0x9B
000679: Oct 14 12:49:37.473 bst: ISAKMP:(0):Encryption algorithm offered does not match policy!
000680: Oct 14 12:49:37.473 bst: ISAKMP:(0):atts are not acceptable. Next payload is 3
000681: Oct 14 12:49:37.473 bst: ISAKMP:(0):Checking ISAKMP transform 1 against priority 1 policy
000682: Oct 14 12:49:37.473 bst: ISAKMP:      keylength of 256
000683: Oct 14 12:49:37.473 bst: ISAKMP:      encryption AES-CBC
000684: Oct 14 12:49:37.473 bst: ISAKMP:      hash MD5
000685: Oct 14 12:49:37.473 bst: ISAKMP:      default group 2
000686: Oct 14 12:49:37.473 bst: ISAKMP:      auth XAUTHInitPreShared
000687: Oct 14 12:49:37.473 bst: ISAKMP:      life type in seconds
000688: Oct 14 12:49:37.473 bst: ISAKMP:      life duration (VPI) of  0x0 0x20 0xC4 0x9B
000689: Oct 14 12:49:37.473 bst: ISAKMP:(0):Encryption algorithm offered does not match policy!
000690: Oct 14 12:49:37.473 bst: ISAKMP:(0):atts are not acceptable. Next payload is 3
000691: Oct 14 12:49:37.473 bst: ISAKMP:(0):Checking ISAKMP transform 2 against priority 1 policy
000692: Oct 14 12:49:37.473 bst: ISAKMP:      keylength of 192
000693: Oct 14 12:49:37.473 bst: ISAKMP:      encryption AES-CBC
000694: Oct 14 12:49:37.473 bst: ISAKMP:      hash SHA
000695: Oct 14 12:49:37.473 bst: ISAKMP:      default group 2
000696: Oct 14 12:49:37.473 bst: ISAKMP:      auth XAUTHInitPreShared
000697: Oct 14 12:49:37.477 bst: ISAKMP:      life type in seconds
000698: Oct 14 12:49:37.477 bst: ISAKMP:      life duration (VPI) of  0x0 0x20 0xC4 0x9B
000699: Oct 14 12:49:37.477 bst: ISAKMP:(0):Encryption algorithm offered does not match policy!
000700: Oct 14 12:49:37.477 bst: ISAKMP:(0):atts are not acceptable. Next payload is 3
000701: Oct 14 12:49:37.477 bst: ISAKMP:(0):Checking ISAKMP transform 3 against priority 1 policy
000702: Oct 14 12:49:37.477 bst: ISAKMP:      keylength of 192
000703: Oct 14 12:49:37.477 bst: ISAKMP:      encryption AES-CBC
000704: Oct 14 12:49:37.477 bst: ISAKMP:      hash MD5
000705: Oct 14 12:49:37.477 bst: ISAKMP:      default group 2
000706: Oct 14 12:49:37.477 bst: ISAKMP:      auth XAUTHInitPreShared
000707: Oct 14 12:49:37.477 bst: ISAKMP:      life type in seconds
000708: Oct 14 12:49:37.477 bst: ISAKMP:      life duration (VPI) of  0x0 0x20 0xC4 0x9B
000709: Oct 14 12:49:37.477 bst: ISAKMP:(0):Encryption algorithm offered does not match policy!
000710: Oct 14 12:49:37.477 bst: ISAKMP:(0):atts are not acceptable. Next payload is 3
000711: Oct 14 12:49:37.477 bst: ISAKMP:(0):Checking ISAKMP transform 4 against priority 1 policy
000712: Oct 14 12:49:37.477 bst: ISAKMP:      keylength of 128
000713: Oct 14 12:49:37.477 bst: ISAKMP:      encryption AES-CBC
000714: Oct 14 12:49:37.477 bst: ISAKMP:      hash SHA
000715: Oct 14 12:49:37.477 bst: ISAKMP:      default group 2
000716: Oct 14 12:49:37.477 bst: ISAKMP:      auth XAUTHInitPreShared
000717: Oct 14 12:49:37.477 bst: ISAKMP:      life type in seconds
000718: Oct 14 12:49:37.477 bst: ISAKMP:      life duration (VPI) of  0x0 0x20 0xC4 0x9B
000719: Oct 14 12:49:37.477 bst: ISAKMP:(0):Encryption algorithm offered does not match policy!
000720: Oct 14 12:49:37.477 bst: ISAKMP:(0):atts are not acceptable. Next payload is 3
000721: Oct 14 12:49:37.477 bst: ISAKMP:(0):Checking ISAKMP transform 5 against priority 1 policy
000722: Oct 14 12:49:37.477 bst: ISAKMP:      keylength of 128
000723: Oct 14 12:49:37.477 bst: ISAKMP:      encryption AES-CBC
000724: Oct 14 12:49:37.477 bst: ISAKMP:      hash MD5
000725: Oct 14 12:49:37.477 bst: ISAKMP:      default group 2
000726: Oct 14 12:49:37.477 bst: ISAKMP:      auth XAUTHInitPreShared
000727: Oct 14 12:49:37.477 bst: ISAKMP:      life type in seconds
000728: Oct 14 12:49:37.477 bst: ISAKMP:      life duration (VPI) of  0x0 0x20 0xC4 0x9B
000729: Oct 14 12:49:37.477 bst: ISAKMP:(0):Encryption algorithm offered does not match policy!
000730: Oct 14 12:49:37.481 bst: ISAKMP:(0):atts are not acceptable. Next payload is 3
000731: Oct 14 12:49:37.481 bst: ISAKMP:(0):Checking ISAKMP transform 6 against priority 1 policy
000732: Oct 14 12:49:37.481 bst: ISAKMP:      encryption 3DES-CBC
000733: Oct 14 12:49:37.481 bst: ISAKMP:      hash SHA
000734: Oct 14 12:49:37.481 bst: ISAKMP:      default group 2
000735: Oct 14 12:49:37.481 bst: ISAKMP:      auth XAUTHInitPreShared
000736: Oct 14 12:49:37.481 bst: ISAKMP:      life type in seconds
000737: Oct 14 12:49:37.481 bst: ISAKMP:      life duration (VPI) of  0x0 0x20 0xC4 0x9B
000738: Oct 14 12:49:37.481 bst: ISAKMP:(0):Hash algorithm offered does not match policy!
000739: Oct 14 12:49:37.481 bst: ISAKMP:(0):atts are not acceptable. Next payload is 3
000740: Oct 14 12:49:37.481 bst: ISAKMP:(0):Checking ISAKMP transform 7 against priority 1 policy
000741: Oct 14 12:49:37.481 bst: ISAKMP:      encryption 3DES-CBC
000742: Oct 14 12:49:37.481 bst: ISAKMP:      hash MD5
000743: Oct 14 12:49:37.481 bst: ISAKMP:      default group 2
000744: Oct 14 12:49:37.481 bst: ISAKMP:      auth XAUTHInitPreShared
000745: Oct 14 12:49:37.481 bst: ISAKMP:      life type in seconds
000746: Oct 14 12:49:37.481 bst: ISAKMP:      life duration (VPI) of  0x0 0x20 0xC4 0x9B
000747: Oct 14 12:49:37.481 bst: ISAKMP:(0):atts are acceptable. Next payload is 3
000748: Oct 14 12:49:37.481 bst: ISAKMP:(0):Acceptable atts:actual life: 86400
000749: Oct 14 12:49:37.481 bst: ISAKMP:(0):Acceptable atts:life: 0
000750: Oct 14 12:49:37.481 bst: ISAKMP:(0):Fill atts in sa vpi_length:4
000751: Oct 14 12:49:37.481 bst: ISAKMP:(0):Fill atts in sa life_in_seconds:2147483
000752: Oct 14 12:49:37.481 bst: ISAKMP:(0):Returning Actual lifetime: 86400
000753: Oct 14 12:49:37.481 bst: ISAKMP:(0)::Started lifetime timer: 86400.

000754: Oct 14 12:49:37.481 bst: ISAKMP:(0): processing KE payload. message ID = 0
000755: Oct 14 12:49:37.525 bst: ISAKMP:(0): processing NONCE payload. message ID = 0
000756: Oct 14 12:49:37.525 bst: ISAKMP (0:0): vendor ID is NAT-T RFC 3947
000757: Oct 14 12:49:37.525 bst: ISAKMP:(0): vendor ID is NAT-T v2
000758: Oct 14 12:49:37.529 bst: ISAKMP:(0):Input = IKE_MESG_FROM_PEER, IKE_AM_EXCH
000759: Oct 14 12:49:37.529 bst: ISAKMP:(0):Old State = IKE_READY  New State = IKE_R_AM_AAA_AWAIT

000760: Oct 14 12:49:37.529 bst: ISAKMP:(2007): constructed NAT-T vendor-rfc3947 ID
000761: Oct 14 12:49:37.529 bst: ISAKMP:(2007):SA is doing pre-shared key authentication plus XAUTH using id type ID_IPV4_ADDR
000762: Oct 14 12:49:37.529 bst: ISAKMP (0:2007): ID payload
        next-payload : 10
        type         : 1
        address      : 123.1.1.185
        protocol     : 17
        port         : 0
        length       : 12
000763: Oct 14 12:49:37.529 bst: ISAKMP:(2007):Total payload length: 12
000764: Oct 14 12:49:37.533 bst: ISAKMP:(2007): sending packet to 120.3.3.142 my_port 500 peer_port 500 (R) AG_INIT_EXCH
000765: Oct 14 12:49:37.533 bst: ISAKMP:(2007):Sending an IKE IPv4 Packet.
000766: Oct 14 12:49:37.533 bst: ISAKMP:(2007):Input = IKE_MESG_FROM_AAA, PRESHARED_KEY_REPLY
000767: Oct 14 12:49:37.533 bst: ISAKMP:(2007):Old State = IKE_R_AM_AAA_AWAIT  New State = IKE_R_AM2

000768: Oct 14 12:49:37.713 bst: ISAKMP (0:2007): received packet from 120.3.3.142 dport 4500 sport 4500 Global (R) AG_INIT_EXCH
000769: Oct 14 12:49:37.717 bst: ISAKMP:(2007): processing HASH payload. message ID = 0
000770: Oct 14 12:49:37.717 bst: ISAKMP:(2007): processing NOTIFY INITIAL_CONTACT protocol 1
        spi 0, message ID = 0, sa = 8210A588
000771: Oct 14 12:49:37.717 bst: ISAKMP:received payload type 20
000772: Oct 14 12:49:37.717 bst: ISAKMP (0:2007): NAT found, the node inside NAT
000773: Oct 14 12:49:37.717 bst: ISAKMP:received payload type 20
000774: Oct 14 12:49:37.717 bst: ISAKMP (0:2007): NAT found, both nodes are all located inside NAT
000775: Oct 14 12:49:37.717 bst: ISAKMP:(2007):SA authentication status:
        authenticated
000776: Oct 14 12:49:37.717 bst: ISAKMP:(2007):SA has been authenticated with 120.3.3.142
000777: Oct 14 12:49:37.717 bst: ISAKMP:(2007):Detected port,floating to port = 4500
000778: Oct 14 12:49:37.717 bst: ISAKMP: Trying to find existing peer 123.1.1.185/120.3.3.142/4500/ and found existing peer 82E01CCC to reuse, free 82F6A850
000779: Oct 14 12:49:37.717 bst: ISAKMP: Unlocking peer struct 0x82F6A850 Reuse existing peer, count 0
000780: Oct 14 12:49:37.717 bst: ISAKMP: Deleting peer node by peer_reap for 120.3.3.142: 82F6A850
000781: Oct 14 12:49:37.717 bst: ISAKMP: Locking peer struct 0x82E01CCC, refcount 2 for Reuse existing peer
000782: Oct 14 12:49:37.717 bst: ISAKMP:(2007):SA authentication status:
        authenticated
000783: Oct 14 12:49:37.717 bst: ISAKMP:(2007): Process initial contact,
bring down existing phase 1 and 2 SA's with local 123.1.1.185 remote 120.3.3.142 remote port 4500
000784: Oct 14 12:49:37.717 bst: ISAKMP:(2007):returning IP addr to the address pool: 10.10.148.11
000785: Oct 14 12:49:37.717 bst: ISAKMP (0:2007): returning address 10.10.148.11 to pool
000786: Oct 14 12:49:37.721 bst: ISAKMP:(2005):received initial contact, deleting SA
000787: Oct 14 12:49:37.721 bst: ISAKMP:(2005):peer does not do paranoid keepalives.

000788: Oct 14 12:49:37.721 bst: ISAKMP:(2005):peer does not do paranoid keepalives.

000789: Oct 14 12:49:37.721 bst: ISAKMP:(2005):deleting SA reason "Receive initial contact" state (R) CONF_XAUTH    (peer 120.3.3.142)
000790: Oct 14 12:49:37.721 bst: ISAKMP:(2007):Setting UDP ENC peer struct 0x0 sa= 0x8210A588
000791: Oct 14 12:49:37.721 bst: ISAKMP:(2007):Returning Actual lifetime: 86400
000792: Oct 14 12:49:37.721 bst: ISAKMP: set new node 38746371 to CONF_XAUTH
000793: Oct 14 12:49:37.721 bst: ISAKMP:(2007):Sending NOTIFY RESPONDER_LIFETIME protocol 1
        spi 2192666280, message ID = 38746371
000794: Oct 14 12:49:37.721 bst: ISAKMP:(2007): sending packet to 120.3.3.142 my_port 4500 peer_port 4500 (R) QM_IDLE
000795: Oct 14 12:49:37.721 bst: ISAKMP:(2007):Sending an IKE IPv4 Packet.
000796: Oct 14 12:49:37.721 bst: ISAKMP:(2007):purging node 38746371
000797: Oct 14 12:49:37.721 bst: ISAKMP: Sending phase 1 responder lifetime 86400

000798: Oct 14 12:49:37.725 bst: ISAKMP:(2007):Input = IKE_MESG_FROM_PEER, IKE_AM_EXCH
000799: Oct 14 12:49:37.725 bst: ISAKMP:(2007):Old State = IKE_R_AM2  New State = IKE_P1_COMPLETE

000800: Oct 14 12:49:37.725 bst: IPSEC(key_engine): got a queue event with 1 KMI message(s)
000801: Oct 14 12:49:37.725 bst: IPSEC(key_engine): got a queue event with 1 KMI message(s)
000802: Oct 14 12:49:37.725 bst: IPSEC(key_engine): got a queue event with 1 KMI message(s)
000803: Oct 14 12:49:37.725 bst: IPSEC(key_engine_delete_sas): rec'd delete notify from ISAKMP
000804: Oct 14 12:49:37.725 bst: IPSEC(key_engine_delete_sas): delete all SAs shared with peer 120.3.3.142
000805: Oct 14 12:49:37.725 bst: ISAKMP: set new node -497880832 to CONF_XAUTH
000806: Oct 14 12:49:37.729 bst: ISAKMP:(2005): sending packet to 120.3.3.142 my_port 4500 peer_port 4500 (R) CONF_XAUTH
000807: Oct 14 12:49:37.729 bst: ISAKMP:(2005):Sending an IKE IPv4 Packet.
000808: Oct 14 12:49:37.729 bst: ISAKMP:(2005):purging node -497880832
000809: Oct 14 12:49:37.729 bst: ISAKMP:(2005):Input = IKE_MESG_INTERNAL, IKE_PHASE1_DEL
000810: Oct 14 12:49:37.729 bst: ISAKMP:(2005):Old State = IKE_XAUTH_REQ_SENT  New State = IKE_DEST_SA

000811: Oct 14 12:49:37.729 bst: ISAKMP:(2007):Need XAUTH
000812: Oct 14 12:49:37.729 bst: ISAKMP: set new node -1159538072 to CONF_XAUTH
000813: Oct 14 12:49:37.729 bst: ISAKMP/xauth: request attribute XAUTH_USER_NAME_V2
000814: Oct 14 12:49:37.729 bst: ISAKMP/xauth: request attribute XAUTH_USER_PASSWORD_V2
000815: Oct 14 12:49:37.729 bst: ISAKMP:(2007): initiating peer config to 120.3.3.142. ID = -1159538072
000816: Oct 14 12:49:37.729 bst: ISAKMP:(2007): sending packet to 120.3.3.142 my_port 4500 peer_port 4500 (R) CONF_XAUTH
000817: Oct 14 12:49:37.729 bst: ISAKMP:(2007):Sending an IKE IPv4 Packet.
000818: Oct 14 12:49:37.733 bst: ISAKMP:(2007):Input = IKE_MESG_INTERNAL, IKE_PHASE1_COMPLETE
000819: Oct 14 12:49:37.733 bst: ISAKMP:(2007):Old State = IKE_P1_COMPLETE  New State = IKE_XAUTH_REQ_SENT

000820: Oct 14 12:49:37.733 bst: ISAKMP:(2005):deleting SA reason "Receive initial contact" state (R) CONF_XAUTH    (peer 120.3.3.142)
000821: Oct 14 12:49:37.733 bst: ISAKMP:(0):Can't decrement IKE Call Admission Control stat incoming_active since it's already 0.
000822: Oct 14 12:49:37.733 bst: ISAKMP: Unlocking peer struct 0x82E01CCC for isadb_mark_sa_deleted(), count 1
000823: Oct 14 12:49:37.733 bst: ISAKMP:(2005):deleting node -565811019 error FALSE reason "IKE deleted"
000824: Oct 14 12:49:37.733 bst: ISAKMP:(2005):Input = IKE_MESG_FROM_PEER, IKE_MM_EXCH
000825: Oct 14 12:49:37.733 bst: ISAKMP:(2005):Old State = IKE_DEST_SA  New State = IKE_DEST_SA

000826: Oct 14 12:49:39.724 bst: ISAKMP (0:2007): received packet from 120.3.3.142 dport 4500 sport 4500 Global (R) CONF_XAUTH
000827: Oct 14 12:49:39.724 bst: ISAKMP:(2007): phase 1 packet is a duplicate of a previous packet.
000828: Oct 14 12:49:39.724 bst: ISAKMP:(2007): retransmitting due to retransmit phase 1
000829: Oct 14 12:49:39.724 bst: ISAKMP:(2007): no outgoing phase 1 packet to retransmit. CONF_XAUTH
000830: Oct 14 12:49:40.248 bst: ISAKMP:(2001):purging SA., sa=82C3A740, delme=82C3A740
000831: Oct 14 12:49:40.248 bst: ISAKMP:(2001):purging node 420788257
000832: Oct 14 12:49:40.248 bst: ISAKMP:(2001):purging node -330594349
000833: Oct 14 12:49:43.734 bst: ISAKMP (0:2007): received packet from 120.3.3.142 dport 4500 sport 4500 Global (R) CONF_XAUTH
000834: Oct 14 12:49:43.734 bst: ISAKMP:(2007): phase 1 packet is a duplicate of a previous packet.
000835: Oct 14 12:49:43.734 bst: ISAKMP:(2007): retransmitting due to retransmit phase 1
000836: Oct 14 12:49:43.734 bst: ISAKMP:(2007): no outgoing phase 1 packet to retransmit. CONF_XAUTH
000837: Oct 14 12:49:45.278 bst: ISAKMP:(2006):purging SA., sa=82F39980, delme=82F39980
000838: Oct 14 12:49:45.278 bst: ISAKMP:(2006):purging node 1765509531
000839: Oct 14 12:49:45.278 bst: ISAKMP:(2006):purging node -1220737993
000840: Oct 14 12:49:51.746 bst: ISAKMP (0:2007): received packet from 120.3.3.142 dport 4500 sport 4500 Global (R) CONF_XAUTH
000841: Oct 14 12:49:51.746 bst: ISAKMP:(2007): phase 1 packet is a duplicate of a previous packet.
000842: Oct 14 12:49:51.746 bst: ISAKMP:(2007): retransmitting due to retransmit phase 1
000843: Oct 14 12:49:51.746 bst: ISAKMP:(2007): no outgoing phase 1 packet to retransmit. CONF_XAUTH

Cisco Employee

Re: VPN return traffic not flowing over the tunnel

Ok, this is getting more and more intriguing

I see from the debugs that vpnc is using AM (agressive mode), and at http://www.unix-ag.uni-kl.de/~massar/vpnc/ I read that it does not support MM (main mode).

Now what should happen with AM, is:

Initiator sends AM1 on UDP 500

Responder sends AM2 on UDP 500

Initiator sends AM3 on UDP 4500

=> phase 1 complete

Responder sends Xauth request on UDP 4500

Initiator sends Xauth response etc. etc.

However here we see:

Initiator sends AM1 on UDP 500 - ok

Responder sends AM2 on UDP 500 - ok

Initiator sends AM3 on UDP 4500 - ok

=> phase 1 complete - ok

Responder sends Xauth request on UDP 4500 - ok

Initiator re-sends AM3 .... not ok

So it looks like the client (initiator) receives AM2 (on port 500) but not the Xauth request (on port 4500).

Another snoop on the client would confirm if it is receiving any UDP 4500 ?

Maybe your router is blocking this port?

Also, if you say you can connect to another VPN router then perhaps snooping a working connection there can learn us more. Are you also using cisco-udp for that connection? Is it also an IOS router you connect to?

Herbert

New Member

Re: VPN return traffic not flowing over the tunnel

Hi Herbert.

When vpnc is configured with force-natt I can see exactly these packets:

10.10.40.106 -> 123.1.1.185 UDP D=500 S=500 LEN=1294

123.1.1.185 -> 10.10.40.106 UDP D=500 S=500 LEN=412

10.10.40.106 -> 123.1.1.185 UDP D=4500 S=4500 LEN=168

10.10.40.106 -> 123.1.1.185 UDP D=4500 S=4500 LEN=168

10.10.40.106 -> 123.1.1.185 UDP D=4500 S=4500 LEN=168

The last packet repeats a few times more and then vpnc gives up.

Interestingly, if I leave the nat option off the vpnc config it doesn't act any differently to the above.

I see nothing in the other Cisco's router that blocks port 4500 anywhere.

When snooping the interface observing a successful connection to my company's VPN, I can see this (IP addresses altered):

10.10.40.106 -> 210.21.21.123 UDP D=500 S=500 LEN=842
210.21.21.123 -> 10.10.40.106 UDP IP fragment ID=151 Offset=0    MF=1 TOS=0x0 TTL=246
210.21.21.123 -> 10.10.40.106 UDP IP fragment ID=151 Offset=1480 MF=0 TOS=0x0 TTL=246
10.10.40.106 -> 210.21.21.123 UDP D=4500 S=4500 LEN=232
210.21.21.123 -> 10.10.40.106 UDP D=4500 S=4500 LEN=88
10.10.40.106 -> 210.21.21.123 UDP D=4500 S=4500 LEN=104
210.21.21.123 -> 10.10.40.106 UDP D=4500 S=4500 LEN=80
10.10.40.106 -> 210.21.21.123 UDP D=4500 S=4500 LEN=80
10.10.40.106 -> 210.21.21.123 UDP D=4500 S=4500 LEN=184
210.21.21.123 -> 10.10.40.106 UDP D=4500 S=4500 LEN=224

There is no option covering NAT in that config file at all. I don't know what the hardware I'm terminating on is, but rumour has it it's CISCO kit, but what it is, I couldn't say (or even find out).

Eric

Cisco Employee

Re: VPN return traffic not flowing over the tunnel

Hi Eric,

When vpnc is configured with force-natt I can see exactly these packets:

10.10.40.106 -> 123.1.1.185 UDP D=500 S=500 LEN=1294

123.1.1.185 -> 10.10.40.106 UDP D=500 S=500 LEN=412

10.10.40.106 -> 123.1.1.185 UDP D=4500 S=4500 LEN=168

10.10.40.106 -> 123.1.1.185 UDP D=4500 S=4500 LEN=168

10.10.40.106 -> 123.1.1.185 UDP D=4500 S=4500 LEN=168

ok so this confirms my suspicion, the VPNrouter sends the

Xauth-request on port 4500 but your client does not see it, so it keeps resending AM3 (and after a while gives up).

When snooping the interface observing a successful connection to my company's VPN, I can see this (IP addresses altered):

10.10.40.106 -> 210.21.21.123 UDP D=500 S=500 LEN=842
210.21.21.123 -> 10.10.40.106 UDP IP fragment ID=151 Offset=0    MF=1 TOS=0x0 TTL=246
210.21.21.123 -> 10.10.40.106 UDP IP fragment ID=151 Offset=1480 MF=0 TOS=0x0 TTL=246
10.10.40.106 -> 210.21.21.123 UDP D=4500 S=4500 LEN=232
210.21.21.123 -> 10.10.40.106 UDP D=4500 S=4500 LEN=88
10.10.40.106 -> 210.21.21.123 UDP D=4500 S=4500 LEN=104
210.21.21.123 -> 10.10.40.106 UDP D=4500 S=4500 LEN=80
10.10.40.106 -> 210.21.21.123 UDP D=4500 S=4500 LEN=80
10.10.40.106 -> 210.21.21.123 UDP D=4500 S=4500 LEN=184
210.21.21.123 -> 10.10.40.106 UDP D=4500 S=4500 LEN=224

Ok so now at least we know that with NAT-T it should work.

So the question is, why do the UDP4500 packets from your company's vpn arrive (cfr the line in bold), but not the ones from your VPNrouter?

I see nothing in the other Cisco's router that blocks port 4500 anywhere.

Which router do you mean ?

I meant on the NAT router that your vpnc is behind. Is it configured with PAT / port forwarding for port 4500 ? I guess it is otherwise the connection to your company wouldn't work. Is there an ACL maybe that allows UDP 4500 from your company but not from the other vpn router?

The only other explanation I can think of is that the ISP that your VPNrouter connects to, drops the UDP 4500...

Herbert

New Member

Re: VPN return traffic not flowing over the tunnel

Ok so now at least we know that with NAT-T it should work.

So the question is, why do the UDP4500 packets from your company's vpn arrive (cfr the line in bold), but not the ones from your VPNrouter?


Do we know though that my vpn gateway is sending it? Is there any way to "snoop" the router's Dialer1 interface?

I see nothing in the other Cisco's router that blocks port 4500 anywhere.

Which router do you mean ?

I meant on the NAT router that your vpnc is behind. Is it configured with PAT / port forwarding for port 4500 ? I guess it is otherwise the connection to your company wouldn't work. Is there an ACL maybe that allows UDP 4500 from your company but not from the other vpn router?

I did mean the same router you are referring to, there is no reference to port 4500 in that router config at all.

The only other explanation I can think of is that the ISP that your VPNrouter connects to, drops the UDP 4500...


Hmm, I can ask them, but given that I can VPN into my company's VPN router from that internet connection, too, I don't think so.

The other option of course still is that I simply need to update the firmware on my VPN gateway/router, but the last time I tried to buy a Cisco support contract, I gave up after several weeks of trying to find someone willing to sell me anything for less than several thousand pound.... :-(

Eric

Cisco Employee

Re: VPN return traffic not flowing over the tunnel

Ok so now at least we know that with NAT-T it should work.

So the question is, why do the UDP4500 packets from your company's vpn arrive (cfr the line in bold), but not the ones from your VPNrouter?


Do we know though that my vpn gateway is sending it? Is there any way to "snoop" the router's Dialer1 interface?

I don't remember if you mentioned the IOS version; if it runs IOS 12.3(4)T or later you could try RITE; I've never used it myself so not sure how well it works on a dialer interface:

http://www.cisco.com/en/US/docs/ios/sec_user_services/configuration/guide/sec_ip_traff_export.html

I did mean the same router you are referring to, there is no reference to port 4500 in that router config at all.

Ok; just to be sure, would you mind posting the config here? Or just the acl and nat?

The only other explanation I can think of is that the ISP that your VPNrouter connects to, drops the UDP 4500...


Hmm, I can ask them, but given that I can VPN into my company's VPN router from that internet connection, too, I don't think so.

If that connection also uses nat-t, then you are right. And I think it is very unlikely anyway that it would drop 4500 and not 500. I only mentioned it because I like to consider all options.

The other option of course still is that I simply need to update the firmware on my VPN gateway/router, but the last time I tried to buy a Cisco support contract, I gave up after several weeks of trying to find someone willing to sell me anything for less than several thousand pound.... :-(

Well, having just said I consider all options I can't fully exclude the vpnrouter as the culprit, however I do think it is very unlikely that the vpnrouter is sending the udp500 packet but not the udp4500 packet.

Herbert

New Member

Re: VPN return traffic not flowing over the tunnel

Ok; just to be sure, would you mind posting the config here? Or just the acl and nat?

Here is the config of the outgoing router:

Current configuration : 2683 bytes
!
! No configuration change since last restart
!
version 12.2
no service pad
service timestamps debug uptime
service timestamps log uptime
service password-encryption
!
hostname cisco-vispa
!
logging queue-limit 100
enable secret 5
!
username password 7
username password 7
clock timezone gmt 0
clock summer-time bst recurring last Sun Mar 1:00 last Sun Oct 1:00
ip subnet-zero
no ip source-route
ip host nfs1 10.10.140.207
ip name-server 208.67.222.222
ip name-server 208.67.220.220
ip dhcp database dhcpinternal
ip dhcp excluded-address 10.10.40.1 10.10.40.99
ip dhcp excluded-address 10.10.40.151 10.10.40.255
!
ip dhcp pool dhcpinternal
   import all
   network 10.10.40.0 255.255.255.0
   default-router 10.10.40.1
   dns-server 208.67.222.222 208.67.220.220
!
!
ip audit notify log
ip audit po max-events 100
no ftp-server write-enable
!
!
!
!
!
!
!
interface Ethernet0
ip address 10.10.40.1 255.255.255.0
ip access-group 121 in
no ip redirects
no ip proxy-arp
ip nat inside
hold-queue 100 out
!
interface ATM0
no ip address
no ip mroute-cache
no atm ilmi-keepalive
pvc 0/38
  encapsulation aal5mux ppp dialer
  dialer pool-member 1
!
dsl operating-mode auto
hold-queue 224 in
!
interface Dialer1
ip address negotiated
ip access-group 121 in
no ip redirects
no ip proxy-arp
ip nat outside
encapsulation ppp
no ip split-horizon
dialer pool 1
dialer idle-timeout 0
dialer persistent
dialer-group 1
ppp authentication chap callin
ppp chap hostname
ppp chap password 7
!
ip nat inside source list 18 interface Dialer1 overload
ip nat inside source static 10.10.40.8 interface Dialer1
ip classless
ip route 0.0.0.0 0.0.0.0 Dialer1
ip route 10.10.50.0 255.255.255.0 10.10.40.2
ip route 10.10.140.0 255.255.255.0 10.10.40.2
no ip http server
no ip http secure-server
!
logging 10.10.140.213
access-list 18 permit any
access-list 121 deny   udp any eq netbios-dgm any
access-list 121 deny   udp any eq netbios-ns any
access-list 121 deny   udp any eq netbios-ss any
access-list 121 deny   tcp any eq 137 any
access-list 121 deny   tcp any eq 138 any
access-list 121 deny   tcp any eq 139 any
access-list 121 permit ip any any
!
line con 0
exec-timeout 120 0
no modem enable
transport preferred none
stopbits 1
line aux 0
stopbits 1
line vty 0 4
access-class 23 in
exec-timeout 120 0
password 7 09610F0A0B0A204A1909
login local
length 0
transport preferred none
!
scheduler max-task-time 5000
sntp server 130.88.202.49
sntp server 130.88.200.98
sntp server 130.88.200.6
sntp server 130.88.203.64
!
end

The only other explanation I can think of is that the ISP that your VPNrouter connects to, drops the UDP 4500...

Hmm, I can ask them, but given that I can VPN into my company's VPN router from that internet connection, too, I don't think so.

If that connection also uses nat-t, then you are right. And I think it is very unlikely anyway that it would drop 4500 and not 500. I only mentioned it because I like to consider all options.


A snoop when connected to my employer's VPN across the same network shows both the UDP 500 packets and UDP 4500 packets aplenty.

Ok so now at least we know that with NAT-T it should work.

So the question is, why do the UDP4500 packets from your company's vpn arrive (cfr the line in bold), but not the ones from your VPNrouter?


Do we know though that my vpn gateway is sending it? Is there any way to "snoop" the router's Dialer1 interface?

I don't remember if you mentioned the IOS version; if it runs IOS 12.3(4)T or later you could try RITE; I've never used it myself so not sure how well it works on a dialer interface:

http://www.cisco.com/en/US/docs/ios/sec_user_services/configuration/guide/sec_ip_traff_export.html

Unfortunately, my 837 doesn't appear to support ip traffic-export :-(

Thanks again for your help.

Eric

Cisco Employee

Re: VPN return traffic not flowing over the tunnel

I'm slowly running out of ideas here I'm afraid...

Maybe instead of RITE you can do:

  access-list 199 permit ip host x.x.x.x host y.y.y.y

where X is the vpnrouter and Y is the client.

then

  debug ip packet detail 199

You could do the same on the NAT router but use access-list 199 permit host x.x.x.x any.

And/or on the NAT router modify the ACL so you have "permit udp host x.x.x.x any eq 4500" before the "permit ip any any" and then check show access-list for any hits.

hth

Herbert

New Member

Re: VPN return traffic not flowing over the tunnel

I'm slowly running out of ideas here I'm afraid...


I'm glad it's not just me... ;-)


Maybe instead of RITE you can do:

  access-list 199 permit ip host x.x.x.x host y.y.y.y

where X is the vpnrouter and Y is the client.

then

  debug ip packet detail 199


That worked! Thanks. I'll have to remember that.

Here's the output:

003661: Oct 15 13:04:50.804 bst: IP: tableid=0, s=120.20.20.142 (Dialer1), d=123.1.1.185 (Dialer1), routed via RIB
003662: Oct 15 13:04:50.804 bst: IP: s=120.20.20.142 (Dialer1), d=123.1.1.185 (Dialer1), len 1314, rcvd 3
003663: Oct 15 13:04:50.804 bst:     UDP src=500, dst=500
003664: Oct 15 13:04:50.852 bst: IP: tableid=0, s=123.1.1.185 (local), d=120.20.20.142 (Dialer1), routed via FIB
003665: Oct 15 13:04:50.852 bst: IP: s=123.1.1.185 (local), d=120.20.20.142 (Dialer1), len 432, sending
003666: Oct 15 13:04:50.852 bst:     UDP src=500, dst=500
003667: Oct 15 13:04:51.036 bst: IP: tableid=0, s=120.20.20.142 (Dialer1), d=123.1.1.185 (Dialer1), routed via RIB
003668: Oct 15 13:04:51.036 bst: IP: s=120.20.20.142 (Dialer1), d=123.1.1.185 (Dialer1), len 188, rcvd 3
003669: Oct 15 13:04:51.036 bst:     UDP src=4500, dst=4500
003670: Oct 15 13:04:51.040 bst: IP: tableid=0, s=123.1.1.185 (local), d=120.20.20.142 (Dialer1), routed via FIB
003671: Oct 15 13:04:51.040 bst: IP: s=123.1.1.185 (local), d=120.20.20.142 (Dialer1), len 124, sending
003672: Oct 15 13:04:51.040 bst:     UDP src=4500, dst=4500
003673: Oct 15 13:04:51.040 bst: IP: tableid=0, s=123.1.1.185 (local), d=120.20.20.142 (Dialer1), routed via FIB
003674: Oct 15 13:04:51.044 bst: IP: s=123.1.1.185 (local), d=120.20.20.142 (Dialer1), len 116, sending
003675: Oct 15 13:04:51.044 bst:     UDP src=4500, dst=4500
003676: Oct 15 13:04:51.044 bst: IP: tableid=0, s=123.1.1.185 (local), d=120.20.20.142 (Dialer1), routed via FIB
003677: Oct 15 13:04:51.044 bst: IP: s=123.1.1.185 (local), d=120.20.20.142 (Dialer1), len 100, sending
003678: Oct 15 13:04:51.044 bst:     UDP src=4500, dst=4500
003679: Oct 15 13:04:53.039 bst: IP: tableid=0, s=120.20.20.142 (Dialer1), d=123.1.1.185 (Dialer1), routed via RIB
003680: Oct 15 13:04:53.039 bst: IP: s=120.20.20.142 (Dialer1), d=123.1.1.185 (Dialer1), len 188, rcvd 3
003681: Oct 15 13:04:53.039 bst:     UDP src=4500, dst=4500
003682: Oct 15 13:04:57.045 bst: IP: tableid=0, s=120.20.20.142 (Dialer1), d=123.1.1.185 (Dialer1), routed via RIB
003683: Oct 15 13:04:57.049 bst: IP: s=120.20.20.142 (Dialer1), d=123.1.1.185 (Dialer1), len 188, rcvd 3
003684: Oct 15 13:04:57.049 bst:     UDP src=4500, dst=4500

We now know that the router is indeed sending the UDP 4500 packet but doing a snoop on the client at the same time, I can't see it arriving.

We also see that the UDP 4500 packet sent from the client arrives.

I'm starting to wonder if I don't have a NAT problem after all, ie. that the 871 that the client is connected to doesn't know where to send the 4500 packet. Unfortunlately the same packet debug doesn't appear to work on the 871. I will have another go....

Thanks again for your help.

Ciao,

Eric

New Member

Re: VPN return traffic not flowing over the tunnel

It seems that the debug ip packet detail on the 837 is a bit, well, buggy (the last post should have read ...doesn't appear to work on the 837....).

The debug setting seems to capture some but not all packets and not repeatably.

I logged a call with my ISP earlier this morning to clarify the UDP port 4500 thing and am waiting for a response.

The not-so-encouraging thing is that I tried via a third ISP (Vodafone USB mobile modem) earlier today with the same results as testing from my second ISP connection: no UDP 4500 reply packet... Having said that of course, Vodafone does also do NAT for their mobile addresses.

I have also noticed something else that's odd: I have a hardware VPN client (for which I can't see any debug output at all) and that's configured to VPN into my company VPN concentrator and it only appears to work on one of my ISP connections, so there is the possibility that both Vodafone and my ISP are doing something "odd" that causes the VPN solution to break.

Either way, I'm down to waiting for my ISP to get back to me now.

Thanks for all your help.

Eric

New Member

Re: VPN return traffic not flowing over the tunnel

My ISP says they don't block anything, which leaves me pretty stuck.

I'm in two different locations early next week and will try a few more ISPs for good measure...

Eric

Cisco Employee

Re: VPN return traffic not flowing over the tunnel

Well, maybe one last thing to try:

on your NAT router (the one your vpnc is behind), change your acl from:

access-list 121 deny   udp any eq netbios-dgm any
access-list 121 deny   udp any eq netbios-ns any
access-list 121 deny   udp any eq netbios-ss any
access-list 121 deny   tcp any eq 137 any
access-list 121 deny   tcp any eq 138 any
access-list 121 deny   tcp any eq 139 any
access-list 121 permit ip any any

to

access-list 121 deny   udp any eq netbios-dgm any
access-list 121 deny   udp any eq netbios-ns any
access-list 121 deny   udp any eq netbios-ss any
access-list 121 deny   tcp any eq 137 any
access-list 121 deny   tcp any eq 138 any
access-list 121 deny   tcp any eq 139 any

access-list 121 permit udp any an eq 4500
access-list 121 permit ip any any

then try to connect and check "show access-list 121" and see if you get any hits on that new line...

BTW which IOS version is it running? I see 12.2 in the config, so not that recent I guess

New Member

Re: VPN return traffic not flowing over the tunnel

That's interesting. The access list shows 7 matches, the debug ip packet detail for the same attempt showed 2 packets (clearly that isn't working properly as I already suspected) and the snoop on the client shows 4 packets.

That would lead me to conclude that the packets do arrive at the NAT router but for some reason the router doesn't know that they ought to be forwarded on to the client. This may be a stupid question: how would the router know where to forward them on to in the first place since we are doing NAT overload? They are after all only UDP packets.

The NAT router's version is:

cisco-vispa#show version
Cisco Internetwork Operating System Software
IOS (tm) C837 Software (C837-K9O3Y6-M), Version 12.2(13)ZH4, EARLY DEPLOYMENT RELEASE SOFTWARE (fc1)
Synched to technology version 12.2(14.5)T
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2004 by cisco Systems, Inc.
Compiled Wed 24-Mar-04 19:00 by ealyon
Image text-base: 0x800131E8, data-base: 0x80AA1DF0

ROM: System Bootstrap, Version 12.2(11r)YV1, RELEASE SOFTWARE (fc1)
ROM: C837 Software (C837-K9O3Y6-M), Version 12.2(13)ZH4, EARLY DEPLOYMENT RELEASE SOFTWARE (fc1)

cisco-vispa uptime is 10 hours, 0 minutes
System returned to ROM by power-on
System restarted at 22:29:46 bst Fri Oct 15 2010
System image file is "flash:c837-k9o3y6-mz.122-13.ZH4.bin"

CISCO C837 (MPC857DSL) processor (revision 0x501) with 44237K/4915K bytes of memory.
Processor board ID FOC08511D6P (1759460522), with hardware revision 0000
CPU rev number 7
Bridging software.
1 Ethernet/IEEE 802.3 interface(s)
1 ATM network interface(s)
128K bytes of non-volatile configuration memory.
12288K bytes of processor board System flash (Read/Write)
2048K bytes of processor board Web flash (Read/Write)

Configuration register is 0x2102

cisco-vispa#

I'll take any hints and tips by the way on where to get a Cisco support agreement that allows me to update the versions on my various pieces of Cisco kit that doesn't cost me thousands of pounds....

Eric

Cisco Employee

Re: VPN return traffic not flowing over the tunnel

Hi Eric

to get a service contract (or maybe just to inquire about the price of an IOS upgrade), check the partner locator tool to fin a Cisco partner near you:

http://tools.cisco.com/WWChannels/LOCATR/openBasicSearch.do

To be honest I have no idea about pricing...

To answer the question about how the router knows how to forward the incoming UDP packets: when the client first sends a UDP packet, the router will create an entry in its NAT table, for the given source & destination address and ports, and set a timer.

When an inbound packet arrives that matches the entry, the packet is "untranslated" and forwarded on the inside (and the timer reset).

If no traffic is seen before the timer expires, the nat entry is removed.

So one thing to check would be "show ip nat trans" (assuming this command exists in your IOS version...) immediately after attempting the vpn connection.

You can also enable "debug ip nat detail" and see if that tells you anything when the UDP4500 packet comes in.

Should you discover that the nat entry times out, then you can tweak the timer with:

basic(config)#ip nat translation udp-timeout ?
  <0-536870>  Timeout in seconds
  never       Never timeout

or:

basic(config)#ip nat translation port-timeout udp 4500 ?
  <0-536870>  Timeout in seconds
  never       Never timeout

(again, not sure if your IOS version already has these commands)

hth

Herbert

New Member

Re: VPN return traffic not flowing over the tunnel

Thanks for the info. I'll try that.

Now back online after two days on the road where I have tried three further public internet connections, same result with all of them. cisco-udp allows the connection but then no return packets flow. force-natt sends the UDP packet on port 4500 but never gets a response.

Note that all three public internet connections were nat'ed. While I think we can rule out that my ISP blocks anything, I don't think we can quite rule out that NAT isn't interfering.

Once I've caught up with all my todos after two days out of the office, I will have another look in detail at your suggestions.

Thanks for bearing with me and for all your good help.

Ciao,

Eric

New Member

Re: VPN return traffic not flowing over the tunnel

My apologies for dragging this out, it's one of those weeks....

I've compared the nat tables of the router to which the client is connected before and after attempting to VPN in. The relevant entries appear to be:

Pro Inside global      Inside local       Outside local      Outside global
udp 123.1.1.185:500  123.1.1.185:500  120.20.20.142:500 120.20.20.142:500
udp 123.1.1.185:1029 123.1.1.185:4500 120.20.20.142:4500 120.20.20.142:4500

One does notice the Inside global port on the second line is 1029 and I'm not clear where that would have come from given the snoop does not show any traffice at all on that port.

The debug at the same time show this:

001247: Oct 22 06:40:49.237 bst: NAT: i: udp (123.1.1.185, 500) -> (120.20.20.142, 500) [36660]
001248: Oct 22 06:40:49.417 bst: NAT: i: udp (123.1.1.185, 4500) -> (120.20.20.142, 4500) [36661]
001249: Oct 22 06:40:49.417 bst: NAT: UDP s=4500->1029, d=4500
001250: Oct 22 06:40:49.421 bst: NAT: i: udp (123.1.1.185, 4500) -> (120.20.20.142, 4500) [36662]
001251: Oct 22 06:40:49.421 bst: NAT: UDP s=4500->1029, d=4500
001252: Oct 22 06:40:49.421 bst: NAT: i: udp (123.1.1.185, 4500) -> (120.20.20.142, 4500) [36663]
001253: Oct 22 06:40:49.421 bst: NAT: UDP s=4500->1029, d=4500

Again, the port 1029 shows up for some reason.

I'm not sure if that is a red herring or if there is an issue with this....

Eric

Cisco Employee

Re: VPN return traffic not flowing over the tunnel

Hi eric,

udp 123.1.1.185:1029 123.1.1.185:4500 120.20.20.142:4500 120.20.20.142:4500


could it be that you incorrectly replace the real addresses with fake ones in the output above?


I.e. normally I would expect something like (if the client has internal ip 10.0.0.1):

udp 123.1.1.185:1029 10.0.0.1:4500 120.20.20.142:4500 120.20.20.142:4500

I.e. the inside local address should be the REAL address of the client, not the public ip address.

Now, about the port 1029, that I would normally say is normal, because this router is doing nat overload (aka dynamic nat or PAT).

So the client sends packets with source port 4500, the router translates the source IP to the dialer interface's address and the source port to a random port.

BUT on the other hand,looking at the debugs you took earlier on the VPN router:

000768: Oct 14 12:49:37.713 bst: ISAKMP (0:2007): received packet from 120.3.3.142 dport 4500 sport 4500 Global (R) AG_INIT_EXCH

we see that it is receiving packets with sport (source port) 4500 ...

Now, these were not taken at the same time, so just to be sure I would suggest to test again and get all the debugs and show commands we collected so far again, on both routers at the same time.

If they confirm the above, then I guess this is a bug in NAT on the  client side router since the debugs say it is translating the source  port but it is not really doing that.

BTW the NAT debugs (or at least the part you copy pasted) only show outbound packets (and indeed you see again "s=4500->1029" which means  it is translating the source port, or at least it should be).

Before the part you quoted, were there debugs about inbound  udp500 packets? (just to make sure there is nothing wrong with the  debugs)

Herbert

Cisco Employee

Re: VPN return traffic not flowing over the tunnel

Thinking about it some more, I had 2 more thoughts:

- my theory doesn't explain why the connection works towards your company vpn

- you could try to add something like this:

ip nat inside source static tcp 10.0.0.1 4500 interface dialer 1 4500

In other words statically nat port 4500, and see if that changes anything.

hth

H

9603
Views
0
Helpful
40
Replies
CreatePlease to create content