×

Warning message

  • Cisco Support Forums is in Read Only mode while the site is being migrated.
  • Cisco Support Forums is in Read Only mode while the site is being migrated.

4 out of 25 VPN tunnel is not getting up.....

Unanswered Question
Mar 20th, 2014
User Badges:

Hi Experts,

I have found one strange problem with IPSec VPN, the scenario is like this, our corporate office is connected to its 25 remote office with IPSec VPN, at corporate site, cisco 2811 router is installed and same type of router is installed at each remote site and IPSec VPN is configured between remote office and corporate office and further each remote site router has two other VPN configured which are working properly. Now the problem is, 4 out of 25 remote offices are not getting up with corporate office, I mean the VPN is not getting up for these location. I sit at corporate office and have tried my level best to up these VPN but the problem not getting resolved.

Now the strange problem is that the VPN gets up by itself, after sometime like in 10days or 20days, for sometime and gets down by itself later.

Anyone who can give some insights where the problem could be and how could i troubleshoot the problem?

Thanks in advance for your valuable response

 

  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 0 (0 ratings)
Loading.
Mike Williams Thu, 03/20/2014 - 20:02
User Badges:
  • Bronze, 100 points or more

Do you have any logs or debugs you could share? What about relevant parts of your config? It's very hard for us to troubleshoot without any information to base our suggestions on.

 

Regards,

Mike

bhuwanchandra6315 Thu, 03/20/2014 - 23:47
User Badges:

Hi Mike,

Here is the relavent configuration at my corporate router, however there are 25 tunnels at my corporate router, only some tunnels are mentiond here:

crypto isakmp policy 10
 hash md5
 authentication pre-share
 group 2

crypto isakmp key My_key address remote_ip1
crypto isakmp key My_key address remote_ip2
crypto isakmp key My_key address remote_ip3
!
!
crypto ipsec transform-set My_transform esp-des esp-md5-hmac
!
crypto map My_map 101 ipsec-isakmp
 set peer remote_ip1
 set transform-set My_transform
 match address 101
crypto map My_map 102 ipsec-isakmp
 set peer remote_ip2
 set transform-set My_transform
 match address 102
crypto map My_map 103 ipsec-isakmp
 set peer remote_ip3
 set transform-set My_transform
 match address 103

!
interface Loopback1
 ip address 172.21.128.1 255.255.255.255
!
interface FastEthernet0/0
 description Towards Internet for VPN
 ip address 10.100.103.2 255.255.255.248
 ip accounting output-packets
 duplex auto
 speed auto
 crypto map My_map
!

!
ip forward-protocol nd
ip route remote_ip1 255.255.255.255 10.100.103.1
ip route remote_ip2 255.255.255.255 10.100.103.1
ip route remote_ip3 255.255.255.255 10.100.103.1
!

access-list 101 permit ip 172.21.128.0 0.0.3.255 172.20.0.0 0.0.31.255
access-list 102 permit ip 172.21.128.0 0.0.3.255 172.20.128.0 0.0.31.255
access-list 103 permit ip 172.21.128.0 0.0.3.255 172.21.158.0 0.0.1.255

Here is the relavent configuration at my remote office router:

crypto isakmp policy 10
 hash md5
 authentication pre-share
 group 2
crypto isakmp key My_key address remote_ip1
crypto isakmp key My_key address remote_ip2
crypto isakmp key My_key address remote_ip3
!
!
crypto ipsec transform-set My_transform esp-des esp-md5-hmac
!
crypto map My_map 1 ipsec-isakmp
 set peer remote_ip1
 set transform-set My_transform
 match address 101
crypto map My_map 2 ipsec-isakmp
 set peer remote_ip2
 set transform-set My_transform
 match address 102
crypto map My_map 3 ipsec-isakmp
 set peer remote_ip3
 set transform-set My_transform
 match address 100
!
!
!
!
!
!
interface FastEthernet0/0
 description Towards Internet for VPN
 ip address 10.100.103.122 255.255.255.248
 duplex auto
 speed auto
 crypto map My_map

interface FastEthernet0/1
 description Towards Local LAN
 ip address 172.21.158.1 255.255.254.0

ip route remote_ip 255.255.255.255 10.100.103.121
ip route remote_ip 255.255.255.255 10.100.103.121
ip route remote_ip 255.255.255.255 10.100.103.121

!
access-list 100 permit ip 172.21.158.0 0.0.1.255 172.20.0.0 0.0.31.255
access-list 101 permit ip 172.21.158.0 0.0.1.255 172.20.128.0 0.0.31.255
access-list 102 permit ip 172.21.158.0 0.0.1.255 172.21.128.0 0.0.3.255
!

here are some logs from corporate router:

Router1#sh crypto isakmp sa
IPv4 Crypto ISAKMP SA
dst             src             state          conn-id slot status
remote_ip   10.100.103.2    QM_IDLE           1198    0 ACTIVE
remote_ip   10.100.103.2    QM_IDLE           1196    0 ACTIVE
remote_ip   10.100.103.2    MM_NO_STATE          0    0 ACTIVE (deleted)

Router1#sh crypto session remote remote_ip
Crypto session current status

Interface: FastEthernet0/0
Session status: DOWN-NEGOTIATING
Peer: remote_ip port 500
  IKE SA: local 10.100.103.2/500 remote remote_ip/500 Inactive
  IPSEC FLOW: permit ip 172.21.128.0/255.255.252.0 172.21.158.0/255.255.254.0
        Active SAs: 0, origin: crypto map

 

here are some logs from remote office router:

Router2#sh crypto ipsec sa peer remote_ip

interface: FastEthernet0/0
    Crypto map tag: My_map , local addr 10.100.103.2

   protected vrf: (none)
   local  ident (addr/mask/prot/port): (172.21.128.0/255.255.252.0/0/0)
   remote ident (addr/mask/prot/port): (172.21.158.0/255.255.254.0/0/0)
   current_peer remote_ip port 500
     PERMIT, flags={origin_is_acl,}
    #pkts encaps: 0, #pkts encrypt: 0, #pkts digest: 0
    #pkts decaps: 0, #pkts decrypt: 0, #pkts verify: 0
    #pkts compressed: 0, #pkts decompressed: 0
    #pkts not compressed: 0, #pkts compr. failed: 0
    #pkts not decompressed: 0, #pkts decompress failed: 0
    #send errors 25, #recv errors 0

     local crypto endpt.: 10.100.103.2, remote crypto endpt.: remote_ip
     path mtu 1500, ip mtu 1500, ip mtu idb FastEthernet0/0
     current outbound spi: 0x0(0)

     inbound esp sas:

     inbound ah sas:

     inbound pcp sas:

     outbound esp sas:

     outbound ah sas:

     outbound pcp sas:

 

router2#sh crypto isakmp sa
IPv4 Crypto ISAKMP SA
dst             src             state          conn-id slot status
remote_ip  10.100.103.122  QM_IDLE           1004    0 ACTIVE
remote_ip  10.100.103.122  QM_IDLE           1003    0 ACTIVE
remote_ip  10.100.103.122  MM_NO_STATE          0    0 ACTIVE
remote_ip  10.100.103.122  MM_NO_STATE          0    0 ACTIVE (deleted)

router2#sh crypto ipsec sa peer remote_ip

interface: FastEthernet0/0
    Crypto map tag: My_map, local addr 10.100.103.122

   protected vrf: (none)
   local  ident (addr/mask/prot/port): (172.21.158.0/255.255.254.0/0/0)
   remote ident (addr/mask/prot/port): (172.21.128.0/255.255.252.0/0/0)
   current_peer remote_ip port 500
     PERMIT, flags={origin_is_acl,ipsec_sa_request_sent}
    #pkts encaps: 0, #pkts encrypt: 0, #pkts digest: 0
    #pkts decaps: 0, #pkts decrypt: 0, #pkts verify: 0
    #pkts compressed: 0, #pkts decompressed: 0
    #pkts not compressed: 0, #pkts compr. failed: 0
    #pkts not decompressed: 0, #pkts decompress failed: 0
    #send errors 6154, #recv errors 0

     local crypto endpt.: 10.100.103.122, remote crypto endpt.: remote_ip 
     path mtu 1500, ip mtu 1500, ip mtu idb FastEthernet0/0
     current outbound spi: 0x0(0)

     inbound esp sas:

     inbound ah sas:

     inbound pcp sas:

     outbound esp sas:

     outbound ah sas:

     outbound pcp sas:

router2#sh crypto session remote remote_ip 
Crypto session current status

Interface: FastEthernet0/0
Session status: DOWN-NEGOTIATING
Peer: remote_ip port 500
  IKE SA: local 10.100.103.122/500 remote remote_ip/500 Inactive
  IKE SA: local 10.100.103.122/500 remote remote_ip/500 Inactive
  IPSEC FLOW: permit ip 172.21.158.0/255.255.254.0 172.21.128.0/255.255.252.0
        Active SAs: 0, origin: crypto map

Based on above config and logs could anyone identify, where the problem could be?

Thanks 

Bhuwan

Mike Williams Fri, 03/21/2014 - 07:50
User Badges:
  • Bronze, 100 points or more

Are you having any internet connectivity issues at the remote sites?

 

Can you capture some debugs?

 

debug crypto condition peer ipv4 *remote peer ip*

debug crypto isakmp

debug crypto engine

debug crypto ipsec

clear log

 

bhuwanchandra6315 Sat, 03/22/2014 - 11:26
User Badges:

Hi Mike,

Thanks for your reply...

Below are some logs from corporate router with one of the tunnel which is not getting up::

RTR-FTR-PJB#debug crypto isakmp
Crypto ISAKMP debugging is on
RTR-FTR-PJB#ping 172.26.10.1 source l1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.26.10.1, timeout is 2 seconds:
Packet sent with a source address of 172.21.128.1

*Mar 22 12:19:32.147: ISAKMP: local port 500, remote port 500
*Mar 22 12:19:32.147: ISAKMP: set new node 0 to QM_IDLE
*Mar 22 12:19:32.147: ISAKMP: Find a dup sa in the avl tree during calling isadb_insert sa = 459BC390
*Mar 22 12:19:32.147: ISAKMP:(0):Can not start Aggressive mode, trying Main mode.
*Mar 22 12:19:32.147: ISAKMP:(0):found peer pre-shared key matching remote_ipsec_peer
*Mar 22 12:19:32.147: ISAKMP:(0): constructed NAT-T vendor-rfc3947 ID
*Mar 22 12:19:32.147: ISAKMP:(0): constructed NAT-T vendor-07 ID
*Mar 22 12:19:32.147: ISAKMP:(0): constructed NAT-T vendor-03 ID
*Mar 22 12:19:32.147: ISAKMP:(0): constructed NAT-T vendor-02 ID
*Mar 22 12:19:32.147: ISAKMP:(0):Input = IKE_MESG_FROM_IPSEC, IKE_SA_REQ_MM
*Mar 22 12:19:32.147: ISAKMP:(0):Old State = IKE_READY  New State = IKE_I_MM1

*Mar 22 12:19:32.147: ISAKMP:(0): beginning Main Mode exchange
*Mar 22 12:19:32.147: ISAKMP:(0): sending packet to remote_ipsec_peer my_port 500 peer_port 500 (I) MM_NO_STATE
*Mar 22 12:19:32.147: ISAKMP:(0):Sending an IKE IPv4 Packet......
Success rate is 0 percent (0/5)
RTR-FTR-PJB#
*Mar 22 12:19:42.147: ISAKMP:(0): retransmitting phase 1 MM_NO_STATE...
*Mar 22 12:19:42.147: ISAKMP (0:0): incrementing error counter on sa, attempt 1 of 5: retransmit phase 1
*Mar 22 12:19:42.147: ISAKMP:(0): retransmitting phase 1 MM_NO_STATE
*Mar 22 12:19:42.147: ISAKMP:(0): sending packet to remote_ipsec_peer my_port 500 peer_port 500 (I) MM_NO_STATE
*Mar 22 12:19:42.147: ISAKMP:(0):Sending an IKE IPv4 Packet.
*Mar 22 12:19:52.147: ISAKMP:(0): retransmitting phase 1 MM_NO_STATE...
*Mar 22 12:19:52.147: ISAKMP (0:0): incrementing error counter on sa, attempt 2 of 5: retransmit phase 1
*Mar 22 12:19:52.147: ISAKMP:(0): retransmitting phase 1 MM_NO_STATE
*Mar 22 12:19:52.147: ISAKMP:(0): sending packet to remote_ipsec_peer my_port 500 peer_port 500 (I) MM_NO_STATE
*Mar 22 12:19:52.147: ISAKMP:(0):Sending an IKE IPv4 Packet.
*Mar 22 12:20:02.143: ISAKMP: set new node 0 to QM_IDLE
*Mar 22 12:20:02.143: ISAKMP:(0):SA is still budding. Attached new ipsec request to it. (local 10.100.103.2, remote remote_ipsec_peer)
*Mar 22 12:20:02.147: ISAKMP:(0): retransmitting phase 1 MM_NO_STATE...
*Mar 22 12:20:02.147: ISAKMP (0:0): incrementing error counter on sa, attempt 3 of 5: retransmit phase 1
*Mar 22 12:20:02.147: ISAKMP:(0): retransmitting phase 1 MM_NO_STATE
*Mar 22 12:20:02.147: ISAKMP:(0): sending packet to remote_ipsec_peer my_port 500 peer_port 500 (I) MM_NO_STATE
*Mar 22 12:20:02.147: ISAKMP:(0):Sending an IKE IPv4 Packet.
*Mar 22 12:20:03.847: ISAKMP:(0):purging node 1974447943
*Mar 22 12:20:03.847: ISAKMP:(0):purging node -1277953536
*Mar 22 12:20:12.147: ISAKMP:(0): retransmitting phase 1 MM_NO_STATE...
*Mar 22 12:20:12.147: ISAKMP (0:0): incrementing error counter on sa, attempt 4 of 5: retransmit phase 1
*Mar 22 12:20:12.147: ISAKMP:(0): retransmitting phase 1 MM_NO_STATE
*Mar 22 12:20:12.147: ISAKMP:(0): sending packet to remote_ipsec_peer my_port 500 peer_port 500 (I) MM_NO_STATE
*Mar 22 12:20:12.147: ISAKMP:(0):Sending an IKE IPv4 Packet.
*Mar 22 12:20:13.847: ISAKMP:(0):purging SA., sa=451DF344, delme=451DF344
*Mar 22 12:20:22.147: ISAKMP:(0): retransmitting phase 1 MM_NO_STATE...
*Mar 22 12:20:22.147: ISAKMP (0:0): incrementing error counter on sa, attempt 5 of 5: retransmit phase 1
*Mar 22 12:20:22.147: ISAKMP:(0): retransmitting phase 1 MM_NO_STATE
*Mar 22 12:20:22.147: ISAKMP:(0): sending packet to remote_ipsec_peermy_port 500 peer_port 500 (I) MM_NO_STATE
*Mar 22 12:20:22.147: ISAKMP:(0):Sending an IKE IPv4 Packet.
*Mar 22 12:20:32.147: ISAKMP:(0): retransmitting phase 1 MM_NO_STATE...
*Mar 22 12:20:32.147: ISAKMP:(0):peer does not do paranoid keepalives.

*Mar 22 12:20:32.147: ISAKMP:(0):deleting SA reason "Death by retransmission P1" state (I) MM_NO_STATE (peer remote_ipsec_peer)
*Mar 22 12:20:32.147: ISAKMP:(0):deleting SA reason "Death by retransmission P1" state (I) MM_NO_STATE (peer remote_ipsec_peer)
*Mar 22 12:20:32.147: ISAKMP:(0):deleting node -1242602279 error FALSE reason "IKE deleted"
*Mar 22 12:20:32.147: ISAKMP:(0):deleting node 275856152 error FALSE reason "IKE deleted"
*Mar 22 12:20:32.147: ISAKMP:(0):Input = IKE_MESG_INTERNAL, IKE_PHASE1_DEL
*Mar 22 12:20:32.147: ISAKMP:(0):Old State = IKE_I_MM1  New State = IKE_DEST_SA

*Mar 22 12:21:22.147: ISAKMP:(0):purging node -1242602279
*Mar 22 12:21:22.147: ISAKMP:(0):purging node 275856152
*Mar 22 12:21:32.147: ISAKMP:(0):purging SA., sa=459BC390, delme=459BC390

 


RTR-FTR-PJB#debug crypto ipsec
Crypto IPSEC debugging is on
RTR-FTR-PJB#ping 172.26.10.1 source l1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.26.10.1, timeout is 2 seconds:
Packet sent with a source address of 172.21.128.1

*Mar 22 12:23:27.411: IPSEC(sa_request): ,
  (key eng. msg.) OUTBOUND local= 10.100.103.2, remote= remote_ipsec_peer,
    local_proxy= 172.21.128.0/255.255.252.0/0/0 (type=4),
    remote_proxy= 172.26.10.0/255.255.254.0/0/0 (type=4),
    protocol= ESP, transform= esp-des esp-md5-hmac  (Tunnel),
    lifedur= 3600s and 4608000kb,
    spi= 0x0(0), conn_id= 0, keysize= 0, flags= 0x0.....
Success rate is 0 percent (0/5)
RTR-FTR-PJB#
*Mar 22 12:23:57.411: IPSEC(key_engine): request timer fired: count = 1,
  (identity) local= 10.100.103.2, remote= remote_ipsec_peer,
    local_proxy= 172.21.128.0/255.255.252.0/0/0 (type=4),
    remote_proxy= 172.26.10.0/255.255.254.0/0/0 (type=4)
*Mar 22 12:23:57.411: IPSEC(sa_request): ,
  (key eng. msg.) OUTBOUND local= 10.100.103.2, remote= remote_ipsec_peer,
    local_proxy= 172.21.128.0/255.255.252.0/0/0 (type=4),
    remote_proxy= 172.26.10.0/255.255.254.0/0/0 (type=4),
    protocol= ESP, transform= esp-des esp-md5-hmac  (Tunnel),
    lifedur= 3600s and 4608000kb,
    spi= 0x0(0), conn_id= 0, keysize= 0, flags= 0x0

RTR-FTR-PJB#debug crypto engine

*Mar 22 12:28:59.415: crypto_engine: Generate IKE hash
*Mar 22 12:28:59.415: crypto_engine: Generate IKE hash
*Mar 22 12:28:59.415: crypto_engine: Encrypt IKE packet
*Mar 22 12:28:59.727: crypto_engine: Generate IKE hash
*Mar 22 12:28:59.727: crypto_engine: Encrypt IKE packet
*Mar 22 12:28:59.763: crypto_engine: Decrypt IKE packet
*Mar 22 12:28:59.763: crypto_engine: Generate IKE hash
*Mar 22 12:29:00.099: crypto_engine: Decrypt IKE packet
*Mar 22 12:29:00.099: crypto_engine: Generate IKE hash
*Mar 22 12:29:00.099: crypto_engine: Generate IKE hash
*Mar 22 12:29:00.099: crypto_engine: Encrypt IKE packet
*Mar 22 12:29:00.239: crypto_engine: Generate IKE hash
*Mar 22 12:29:00.239: crypto_engine: Encrypt IKE packet
*Mar 22 12:29:00.271: crypto_engine: Decrypt IKE packet
*Mar 22 12:29:00.271: crypto_engine: Generate IKE hash
*Mar 22 12:29:00.359: crypto_engine: Decrypt IKE packet
*Mar 22 12:29:00.359: crypto_engine: Generate IKE hash
*Mar 22 12:29:00.359: crypto_engine: Generate IKE hash
*Mar 22 12:29:00.363: crypto_engine: Encrypt IKE packet
*Mar 22 12:29:00.403: crypto_engine: Generate IKE hash

Few things i would like mention here are:

1. I am able to ping remote_ipsec_peer from my router.

2. At both routers other tunnels are working fine.

3. NATing is not involved at both sides router, we have static ip at both side and static routes are configured to reach the peer.

Anyone who can provide some insights by looking the above log, where the problem could be?

Mike Williams Mon, 03/24/2014 - 06:44
User Badges:
  • Bronze, 100 points or more

It appears that router is sending requests to the remote router to establish an ISAKMP session, but not receiving a response to those requests. Can you run the same debugs from the remote router? 

Have you tried rebooting the routers for good measure? Could be a bug.

Regards,

Mike

bhuwanchandra6315 Thu, 03/27/2014 - 23:18
User Badges:

Router has been reloaded lot of time but no luck and i am not able to the check the debug status from remote router.....

The ipsec vpn was working fine till we have replaced the old ADSL modem with new FTTH modem with new static ip. once the modem and static ip got changed the tunnel never came up, however the other two tunnel got up without any problem.

now i want to the answer of few question?

1. Does the new modem have some problem which is making tunnel down like MTU setting, NAT setting?

2. May be the port 500 is blocked somewhere between ISP devices, in that case how we can check the port staus?

Mike Williams Fri, 03/28/2014 - 07:27
User Badges:
  • Bronze, 100 points or more

That helps a lot. It's very likely that if the DSL and IP were the only things that were changed, then it's related to one of those. I assume you have verified your peer addresses are correct on both ends. I have a couple questions:

1. Are the DSL modems in bridging mode, gateway mode, or PPPoE/PPPoA mode?

2. Are the DSL modems doing any NAT?

If they are in bridging mode, they shouldn't be blocking anything. If they are in gateway mode (block of IPs) or PPPoE/A mode, then they may be blocking IPsec on the integrated firewall. 

If you are doing PPPoE/A on the modem, and passing private IP addressing through to the router, then you may need to make some tweaks on the hub router so it uses the private IP of the remote router as the remote router ID. Also, if it's in PPPoE/A mode or bridge mode doing PPPoE/A, then you may need to lower the MTU by 8 bytes on the router, "ip tcp adjust-mss 1492". 

Regards,

Mike

 

bhuwanchandra6315 Sat, 03/29/2014 - 00:33
User Badges:

Thanks mike for your valuable points!!!

Peer address are correctly configured at both the ends. i would like to answer your question:

1. DSL/FTTH modem is in PPPoE mode. The modem is a FTTH modem in which FIber is terminated at on end and one of its ethernet port is connected to router.

2. Yes NAT option is enable in modems WAN tab and NAT type is NAPT. 

3. As you said we are passing private ip to router, means modem's LAN option and router's connected ethernet port is in same subnet as it is quite obvious.

As i already said the other tunnel are working properly at both end, what should i do?. should i lower the mtu size as you said or somthing else....?

Thanks in advance for you support!!!

syed kazim abbas Mon, 04/21/2014 - 23:25
User Badges:
  • Bronze, 100 points or more

Hi,

its a trick may be work, in my case it solved. 

when u make any changing in configuration then do this on both sides.

interface FastEthernet0/0
 description Towards Internet for VPN
 ip address 10.100.103.2 255.255.255.248
 no crypto map My_map 

clear crypto isakmp  1001 (connection-id)

crypto map My-map

 

 

Actions

This Discussion