VPN CONNECTION UP BUT SYSTEMS CANNOT PING EACH OTHER

tomocisco · ‎07-23-2012

Hi All,

Just want to say a big thank you to all the Net Pros that devote their time & resources to help others.

I was recently on this forum when I configured VPN between two of our sites. I got the VPN up and running and all the resources working without hitches. How ever I had to add configure a fail over (Backup) link at the Head office, for this we are using BGP.

The fail over was configured and is running, all the systems are on internet. The VPN is showing UP and active but the only problem is that the internal systems cannot ping across the vpn to reach remote systems. So they can reach any resource on the LAN.

Pasted below is the sho run and show crypto session.

#sho run

Building configuration...

Current configuration : 6144 bytes

!

version 12.4

no service pad

service tcp-keepalives-in

service tcp-keepalives-out

service timestamps debug datetime msec localtime show-timezone

service timestamps log datetime msec localtime show-timezone

service password-encryption

service sequence-numbers

no service dhcp

!

hostname lagos

!

boot-start-marker

boot-end-marker

!

security authentication failure rate 10 log

security passwords min-length 6

logging buffered 4096

logging console critical

enable secret 5

enable password

!

aaa new-model

!

aaa authentication login local_auth local

!

aaa session-id common

!

crypto pki trustpoint TP-self-signed-3885639516

enrollment selfsigned

subject-name cn=IOS-Self-Signed-Certificate-3885639516

revocation-check none

rsakeypair TP-self-signed-3885639516

!

crypto pki certificate chain TP-self-signed-3885639516

certificate self-signed 01

dot11 syslog

no ip source-route

no ip gratuitous-arps

!

ip cef

ip dhcp bootp ignore

!

no ip bootp server

ip domain name

ip name-server 19.x.x.x

ip name-server 4.x.x.x

ip name-server 1.x.x.x

login block-for 15 attempts 5 within 5

!

multilink bundle-name authenticated

password encryption aes

!

username td privilege 15 secret xx

username tdf privilege 15 secret xx

!

crypto isakmp policy 1

encr aes

hash md5

authentication pre-share

group 2

crypto isakmp key xxxx address 4.2.2.2

!

crypto ipsec transform-set ME-VPN esp-aes esp-md5-hmac

!

crypto map VPN-TO-PH 10 ipsec-isakmp

set peer 4.2.2.2

set transform-set ME-VPN

match address VPN-TRAFFIC

!

archive

log config

hidekeys

!

interface Loopback0

ip address 192.200.200.1 255.255.255.255

!

interface Tunnel0

no ip address

!

interface BRI0

no ip address

encapsulation hdlc

shutdown

!

interface FastEthernet0

ip address 192.168.1.1 255.255.255.0

ip nat inside

ip virtual-reassembly

duplex auto

speed auto

!

interface FastEthernet1

ip address 4.2.2.2 255.255.255.248

ip nat outside

ip virtual-reassembly

duplex auto

speed auto

crypto map VPN-TO-PH

!

interface FastEthernet2

!

interface FastEthernet3

!

interface Vlan1

no ip address

!

interface Dialer1

no ip address

!

ip forward-protocol nd

ip route 0.0.0.0 0.0.0.0 4.2.2.1

ip http server

ip http authentication local

ip http secure-server

ip nat inside source route-map SDM_RMAP_1 interface FastEthern

!

ip access-list extended SDM_1

remark SDM_ACL Category=20

permit ip 192.168.1.0 0.0.0.255 192.168.0.0 0.0.0.255

!

access-list 100 remark EXCLUDED FROM NAT

access-list 100 remark SDM_ACL Category=16

access-list 100 deny ip 192.168.1.0 0.0.0.255 192.168.0.0 0.

access-list 100 permit ip 192.168.1.0 0.0.0.255 any

!

route-map SDM_RMAP_1 permit 1

match ip address 100

!

end

Thanks.

Tom

tomocisco · ‎07-24-2012

Hi Giuseppe,

I just thought i should add this observation.

I did a vpn tunnel test from sdm. Though the test was successful but i received this message which I feel may be the reason for the IP connectivity problem:

A ping with data size of this VPN interface MTU size and 'Do not Fragment' bit set to the other end VPN device is failing. This may happen if there is a lesser MTU network which drops the 'Do not fragment' packets.

1)Contact your ISP/Administrator to resolve this issue. 2)Issue the command 'crypto ipsec df-bit clear' under the VPN interface to avoid packets drop due to fragmentation.

I have issued 'crypto ipsec df-bit clear' on both local & remote routers but still no noticable change. The IP connectivity issues still persists.

Thanks

Tom

Giuseppe Larosa · ‎07-24-2012

Hello Tom,

we should understand the issue better.

From router of site1 ( the one speaking eBGP) if you try to ping 192.168.1.1 from 192.168.0.1 what is the result?( I mean using an extended ping)

What is the output of show crypto ipsec sa?

The MTU issue has to be handled later if it is real. But for the moment it is important to understand the connectivity issue.

I remember you had some issues with some static NAT statements.

.

Hope to help

Giuseppe

tomocisco · ‎07-24-2012

Hi Giuseppe,

From the eBGP speaking router extended ping to remote router's internal and external interfaces gives 100% replies

#ping 192.168.1.1 source 192.168.0.1

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 192.168.1.1, timeout is 2 seconds:

Packet sent with a source address of 192.168.0.1

!!!!!

Success rate is 100 percent (5/5), round-trip min/avg/max = 124/304/496 ms

#ping 4.2.2.2 source 192.168.0.1

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 4.2.2.2, timeout is 2 seconds:

Packet sent with a source address of 192.168.0.1

!!!!!

Success rate is 100 percent (5/5), round-trip min/avg/max = 272/409/624 ms

From the non BGP speaking router, extended ping to this side also gives 100% replies when I use source IP as the local internal IP address but gives no reply when I use the public IP of external interface as the source IP.

#ping 192.168.0.1 source 4.2.2.2

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 192.168.0.1, timeout is 2 seconds:

Packet sent with a source address of 4.2.2.2

.....

Success rate is 0 percent (0/5)

#ping 192.168.0.1 source 192.168.1.1

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 192.168.0.1, timeout is 2 seconds:

Packet sent with a source address of 192.168.1.1

!!!!!

Success rate is 100 percent (5/5), round-trip min/avg/max = 116/146/180 ms

#ping 192.168.0.1 source 192.168.1.1

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 192.168.0.1, timeout is 2 seconds:

Packet sent with a source address of 192.168.1.1

!!!!!

Success rate is 100 percent (5/5), round-trip min/avg/max = 144/280/428 ms

Yes its true I had some issues with static NAT statements previously but I removed the static NAT entirely. As u can see from the config earlier posted, the only NAT statement is that which I use for internet access and it is overloading on the Loopback1 interface.

I have a feeling it has to do with NATing but I can't pin point how else to issue the NAT statement since I didnt make any much change.

I can recall however that at the point of configuring the backup I had problem with ineternet connectivity until I added 'IP NAT OUTSIDE' to the two physical interfaces (Fa0 & Fa1) - Primary & Backup links. Initially I had the 'IP NAT OUTSIDE' statement only on the Loopback1 interface.

Thanks

Tom

Giuseppe Larosa · ‎07-24-2012

Hello Tom,

the correct test is to ping internal to internal and this works. (LAN to LAN) you should see the counters increasing in show crypto ipsec sa for both encrypted and decrypted packets and bytes.

What happens if you start the ping from an host in 192.168.0.0/24 and you try to ping an host in 192.168.1.0/24?

I remember you had strange results with 50% of packet loss.

I see that you have removed all the static NAT statements.

Hope to help

Giuseppe

tomocisco · ‎07-24-2012

Hi,

Sorry I forgot to paste the output of the 'sho crypto ipsec sa' It is shown below:

For remote router (non BGP speaking router)

# sho crypto ipsec sa

interface: FastEthernet1

Crypto map tag: VPN-TO-PH, local addr 4.2.2.2

protected vrf: (none)

local ident (addr/mask/prot/port): (192.168.1.0/255.255.255.0/0/0)

remote ident (addr/mask/prot/port): (192.168.0.0/255.255.255.0/0/0)

current_peer 1.2.2.225 port 500

PERMIT, flags={origin_is_acl,}

#pkts encaps: 19473, #pkts encrypt: 19473, #pkts digest: 19473

#pkts decaps: 10, #pkts decrypt: 10, #pkts verify: 10

#pkts compressed: 0, #pkts decompressed: 0

#pkts not compressed: 0, #pkts compr. failed: 0

#pkts not decompressed: 0, #pkts decompress failed: 0

#send errors 1, #recv errors 0

local crypto endpt.: 4.2.2.2, remote crypto endpt.: 1.2.2.225

path mtu 1500, ip mtu 1500, ip mtu idb FastEthernet1

current outbound spi: 0x8C6F3BB5(2356100021)

inbound esp sas:

spi: 0x21352D5D(557133149)

transform: esp-aes esp-md5-hmac ,

in use settings ={Tunnel, }

conn id: 19, flow_id: Motorola SEC 2.0:19, crypto map: VPN-TO-PH

sa timing: remaining key lifetime (k/sec): (4397342/1653)

IV size: 16 bytes

replay detection support: Y

Status: ACTIVE

inbound ah sas:

inbound pcp sas:

outbound esp sas:

spi: 0x8C6F3BB5(2356100021)

transform: esp-aes esp-md5-hmac ,

in use settings ={Tunnel, }

conn id: 20, flow_id: Motorola SEC 2.0:20, crypto map: VPN-TO-PH

sa timing: remaining key lifetime (k/sec): (4397199/1653)

IV size: 16 bytes

replay detection support: Y

Status: ACTIVE

outbound ah sas:

outbound pcp sas:

For Head office router (eBGP speaking router)

#sho crypto ipsec sa

interface: Loopback1

Crypto map tag: VPN-TO-PH, local addr 1.2.2.225

protected vrf: (none)

local ident (addr/mask/prot/port): (0.0.0.0/0.0.0.0/6/179)

remote ident (addr/mask/prot/port): (0.0.0.0/0.0.0.0/6/0)

current_peer 4.2.2.2 port 500

PERMIT, flags={origin_is_acl,}

#pkts encaps: 0, #pkts encrypt: 0, #pkts digest: 0

#pkts decaps: 0, #pkts decrypt: 0, #pkts verify: 0

#pkts compressed: 0, #pkts decompressed: 0

#pkts not compressed: 0, #pkts compr. failed: 0

#pkts not decompressed: 0, #pkts decompress failed: 0

#send errors 0, #recv errors 0

local crypto endpt.: 1.2.2.225, remote crypto endpt.: 4.2.2.2

path mtu 1514, ip mtu 1514, ip mtu idb Loopback1

current outbound spi: 0x0(0)

inbound esp sas:

inbound ah sas:

inbound pcp sas:

outbound esp sas:

outbound ah sas:

outbound pcp sas:

protected vrf: (none)

local ident (addr/mask/prot/port): (0.0.0.0/0.0.0.0/6/0)

remote ident (addr/mask/prot/port): (0.0.0.0/0.0.0.0/6/179)

current_peer 4.2.2.2 port 500

PERMIT, flags={origin_is_acl,}

#pkts encaps: 0, #pkts encrypt: 0, #pkts digest: 0

#pkts decaps: 0, #pkts decrypt: 0, #pkts verify: 0

#pkts compressed: 0, #pkts decompressed: 0

#pkts not compressed: 0, #pkts compr. failed: 0

#pkts not decompressed: 0, #pkts decompress failed: 0

#send errors 0, #recv errors 0

local crypto endpt.: 1.2.2.225, remote crypto endpt.: 4.2.2.2

path mtu 1514, ip mtu 1514, ip mtu idb Loopback1

current outbound spi: 0x0(0)

inbound esp sas:

inbound ah sas:

inbound pcp sas:

outbound esp sas:

outbound ah sas:

outbound pcp sas:

protected vrf: (none)

local ident (addr/mask/prot/port): (192.168.0.0/255.255.255.0/0/0)

remote ident (addr/mask/prot/port): (192.168.1.0/255.255.255.0/0/0)

current_peer 4.2.2.2 port 500

PERMIT, flags={origin_is_acl,}

#pkts encaps: 15, #pkts encrypt: 15, #pkts digest: 15

#pkts decaps: 20462, #pkts decrypt: 20462, #pkts verify: 20462

#pkts compressed: 0, #pkts decompressed: 0

#pkts not compressed: 0, #pkts compr. failed: 0

#pkts not decompressed: 0, #pkts decompress failed: 0

#send errors 0, #recv errors 1

local crypto endpt.: 1.2.2.225, remote crypto endpt.: 4.2.2.2

path mtu 1514, ip mtu 1514, ip mtu idb Loopback1

current outbound spi: 0xD069D21(218537249)

inbound esp sas:

spi: 0xEB28F588(3945330056)

transform: esp-aes esp-md5-hmac ,

in use settings ={Tunnel, }

conn id: 23, flow_id: Motorola SEC 2.0:23, crypto map: VPN-TO-PH

sa timing: remaining key lifetime (k/sec): (4471105/2648)

IV size: 16 bytes

replay detection support: Y

Status: ACTIVE

inbound ah sas:

inbound pcp sas:

outbound esp sas:

spi: 0xD069D21(218537249)

transform: esp-aes esp-md5-hmac ,

in use settings ={Tunnel, }

conn id: 24, flow_id: Motorola SEC 2.0:24, crypto map: VPN-TO-PH

sa timing: remaining key lifetime (k/sec): (4471174/2648)

IV size: 16 bytes

replay detection support: Y

Status: ACTIVE

outbound ah sas:

outbound pcp sas:

The MTU size on one side is 1500 while on the other side it is 1514.

Thanks

Tom

Giuseppe Larosa · ‎07-24-2012

Hello Tom,

these show crypto ipsec sa look like fine

>> local ident (addr/mask/prot/port): (192.168.0.0/255.255.255.0/0/0)

remote ident (addr/mask/prot/port): (192.168.1.0/255.255.255.0/0/0)

current_peer 4.2.2.2 port 500

PERMIT, flags={origin_is_acl,}

>> #pkts encaps: 15, #pkts encrypt: 15, #pkts digest: 15

>> #pkts decaps: 20462, #pkts decrypt: 20462, #pkts verify: 20462

at each ping attempt LAN to LAN the pkts encaps should increase by 5. To be noted the remote end has had less configuration changes and the counter of received and decapsulated = decrypted packets is bigger.

I would try to use an extended ping to send 100 ICMP echo request from HQ router and then check again the counters to see if they are incremented by 100 in pkts encaps line

Hope to help

Giuseppe

Georg Pauwen · ‎07-23-2012

In order to override the eBGP default TTL of 1, try to add the command:

neighbor 17.1.2.1 ebgp-multihop 255

on both routers for the neighbor statements (with the respective IP address of the neighbor on the other side).

If that doesn't help, can you post the output of the 'show ip bgp sum' and 'show ip route bgp' from both routers ?

Regards,

Georg

tomocisco · ‎07-24-2012

Hi Georg,

I can only add the

neighbor 17.1.2.1 ebgp-multihop 255

on my router since i dont have access to the ISP router. Will adding this only on my router make any difference?

The BGP is running between my router and the ISP router (my gateway) but my remote site router is not running BGP, its accessing the net via default route. And the remote site is not presently having a backup (failover) link.

I dont think that the issue is with the remote site router because the major change done was the introduction of BGP at the head office. The only thing done at the remote site was to change the peer address. And the show crypto session will reveal that VPN is ok.

#sho crypto session

Crypto session current status

Interface: Loopback1

Session status: UP-ACTIVE

Peer: 4.2.2.2 port 500

IKE SA: local 19.2.22.22/500 remote 4.2.2.2/500 Active

IPSEC FLOW: permit ip 192.168.0.0/255.255.255.0 192.168.1.0/255.255.255.0

Active SAs: 2, origin: crypto map

Below is the output of sho ip bgp sum

sho ip bgp sum

BGP router identifier 19.2.22.22, local AS number 65142

BGP table version is 10, main routing table version 10

9 network entries using 1080 bytes of memory

17 path entries using 884 bytes of memory

7/6 BGP path/bestpath attribute entries using 868 bytes of memory

5 BGP AS-PATH entries using 120 bytes of memory

0 BGP route-map cache entries using 0 bytes of memory

0 BGP filter-list cache entries using 0 bytes of memory

Bitfield cache entries: current 1 (at peak 1) using 32 bytes of memory

BGP using 2984 total bytes of memory

BGP activity 9/0 prefixes, 17/0 paths, scan interval 60 secs

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd

17.1.2.1 4 65136 341 341 10 0 0 01:32:01 8

17.1.4.1 4 65136 341 341 10 0 0 01:31:58 8

The BGP protocol seems ok. We have good connectivity to the internet and the vpn is up. But the only issue is IP connectivity of the end system thru the vpn connection. What do you think?

Thanks.

Tom

tomocisco · ‎07-24-2012

Hi Georg,

Thought I should add this:

I did a vpn tunnel test from sdm. Though the test was successful but i received this message which I feel may be the reason for the IP connectivity problem:

A ping with data size of this VPN interface MTU size and 'Do not Fragment' bit set to the other end VPN device is failing. This may happen if there is a lesser MTU network which drops the 'Do not fragment' packets.

1)Contact your ISP/Administrator to resolve this issue. 2)Issue the command 'crypto ipsec df-bit clear' under the VPN interface to avoid packets drop due to fragmentation.

I have issued 'crypto ipsec df-bit clear' on both local & remote routers but still no noticable change. The IP connectivity issues still persists.

Thanks

Tom

tomocisco · ‎07-24-2012

Hi All,

I did a vpn tunnel test from sdm. Though the test was successful but i received this message which I feel may be the reason for the IP connectivity problem:

A ping with data size of this VPN interface MTU size and 'Do not Fragment' bit set to the other end VPN device is failing. This may happen if there is a lesser MTU network which drops the 'Do not fragment' packets.

1)Contact your ISP/Administrator to resolve this issue. 2)Issue the command 'crypto ipsec df-bit clear' under the VPN interface to avoid packets drop due to fragmentation.

I have issued 'crypto ipsec df-bit clear' on both local & remote routers but still no noticable change. The IP connectivity issues still persists.

Thanks

Tom

Georg Pauwen · ‎07-24-2012

Tom,

looking over your initial config, I think you might need to allow TCP port 179 (which is used by BGP) to your VPN-TRAFFIC extended access list. Add the following lines and see what happens:

permit tcp any any eq 179

permit tcp any eq 179 any

Regards,

Georg

tomocisco · ‎07-24-2012

Hi Georg,

I added the permit statement to the vpn traffic access list but still no change.

ip access-list extended VPN-TRAFFIC

permit ip 192.168.0.0 0.0.0.255 192.168.1.0 0.0.0.255

permit tcp any any eq bgp

permit tcp any eq bgp any

There is no change to the result.

Thanks for your input.

Tom