IPsec clients can not ping internal hosts

Answered Question
Dec 11th, 2008

Hi, I am setting a new branch office, IPsec L2L tunnel is set between branch ASA 5505 and HQ ASA 5520. HQ ASA is the gateway to software IPsec clients and ezvpn clients operating in network extension mode. HQ (and software IPsec clients) is using 10.0.0.0/8 address space, while ezvpn clients and this branch office will be using 172.16.0.0/12 address space. Branch office uses 172.30.0.0/20 exclusively.

I also configured IPsec RA on branch office, the RA pool is 172.30.1.0/24, full mesh IP connectivity is achieved except that IPsec clients from branch office can not ping hosts in branch office but can ping anywhere else. Specifically, branch office IPsec client is getting ip address 172.30.1.1 and there is a live host in branch office with ip address 172.30.0.253 in inside VLAN, debug capture on ASA5505 of ping from 172.30.1.1 shows that 172.30.0.253 received echo request from 172.30.1.1 and ASA received echo reply from inside VLAN.

This really puzzles me, since ASA5505 has IPsec SA for remote access client, and 172.30.1.1/32 is in the routing table, ASA should simply look at security policy database, and sent the echo reply to the right IPsec peer.

I do have a rather loose access-list defined for L2L ipsec tunnel,

access-list traffic_to_HQ extended permit ip 172.30.0.0 255.255.240.0 10.0.0.0 255.0.0.0

access-list traffic_to_HQ extended permit ip 172.30.0.0 255.255.240.0 172.16.0.0 255.240.0.0

I am wondering maybe the echo-reply is being sent to L2L tunnel because the traffic matches the access-list. But due to longest match rule, IPsec should not use L2L SA to send a packet destined to 172.30.1.1 to L2L tunnel, correct? is there any way to know where the echo reply packets go? any other configuration I might have missed?

Thanks a lot.

Attachment: 
I have this problem too.
0 votes
Correct Answer by husycisco about 7 years 11 months ago

"permit ip any any did not help, I didn't think it would help because the traffic is coming from IPsec tunnel"

My intend was "run packet-tracer again after permitting any to any at outside interface"

Assuming that you have logging enabled, what syslog messages show up when you try to telnet into that router?

Can you create a tunnel-group test and assign a pool which is not covered by any of your inside networks, then create exempt nat and split tunnel acls accordingly? Are results the same in test group with new pool?

Also try

no crypto isakmp nat-traversal 10

crypto isakmp nat-traversal 30

  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 5 (2 ratings)
Loading.
husycisco Fri, 12/12/2008 - 12:40

Hello Jian,

Try adding the following

access-list no-nat extended permit ip 172.30.0.0 255.255.240.0 172.30.1.0 255.255.255.0

Regards

jiangu Fri, 12/12/2008 - 13:36

It did not help ... My no-nat ACL is already covering the traffic from inside network to client network.

husycisco Fri, 12/12/2008 - 14:59

I saw that, its complicated when typing only, a diagram would be helpful if there are any. Can you tell me which IP address you can ping at which interface and you cant ping at which interface while VPN client acquires which IP?

JORGE RODRIGUEZ Fri, 12/12/2008 - 15:32

Hi friend Huseyin,

I was looking at this post earlier but got busy, I did have same thought you did in the nat exempt acl permiting RA vpn pool net to inside net 30.1.0/24 but I guess that did not work , yes a diagram would be helpfull to see since seems to many ipsec points.. but your suggested acl should have worked if ra client net tryiing to access inside net off inside interface..hmmm..puzzle!

Jian,

In adition to a diagram requested by Husseyin and his question, you may also try a low level debug process for icmp , and post output... or u may want to use packet-tracer too.. but icmp debug should give us some hints.

asa(config)#terminal monitor

asa(config)#debug icmp trace

then try a ping from a RA vpn client host to a destination host on the inside 172.30.0.0/24 net.

HTH

Jorge

jiangu Fri, 12/12/2008 - 16:11

Hi, Thanks a lot for your respond, the setup is simple:

172.30.0.253 --- 172.30.0.254[ASA]---Internet--RA client (& HQ-ASA)

Ping capture shows that echo reply is received

ASA# sh capture ping_capture

3 packets captured

1: 17:02:56.616819 802.1Q vlan#5 P0 172.30.0.253 > 172.30.1.1: icmp: echo reply

2: 17:03:01.220493 802.1Q vlan#5 P0 172.30.0.253 > 172.30.1.1: icmp: echo reply

3: 17:03:06.223651 802.1Q vlan#5 P0 172.30.0.253 > 172.30.1.1: icmp: echo reply

But debug ICMP trace did turn out something very interesting:

ICMP echo request from outside:172.30.1.1 to inside:172.30.0.253 ID=1 seq=80 len=32

ICMP echo reply from inside:172.30.0.253 to outside:172.30.1.1 ID=1 seq=80 len=32

ICMP echo reply from outside:172.30.0.253 to outside:172.30.1.1 ID=1 seq=80 len=32

ICMP echo reply from outside:172.30.0.253 to outside:172.30.1.1 ID=1 seq=80 len=32

ICMP echo reply from outside:172.30.0.253 to outside:172.30.1.1 ID=1 seq=80 len=32

ICMP echo reply from outside:172.30.0.253 to outside:172.30.1.1 ID=1 seq=80 len=32

ICMP echo reply from outside:172.30.0.253 to outside:172.30.1.1 ID=1 seq=80 len=32

ICMP echo reply from outside:172.30.0.253 to outside:172.30.1.1 ID=1 seq=80 len=32

Look at first two lines, it is normal, but afterward, the echo reply is looping back from outside interface.

To compare, if I IPsec VPN to corp ASA and then ping 172.30.0.253:

ICMP echo request from outside:172.18.1.1 to inside:172.30.0.253 ID=3 seq=0 len=72

ICMP echo reply from inside:172.30.0.253 to outside:172.18.1.1 ID=3 seq=0 len=72

ICMP echo request from outside:172.18.1.1 to inside:172.30.0.253 ID=3 seq=1 len=72

ICMP echo reply from inside:172.30.0.253 to outside:172.18.1.1 ID=3 seq=1 len=72

ICMP echo request from outside:172.18.1.1 to inside:172.30.0.253 ID=3 seq=2 len=72

ICMP echo reply from inside:172.30.0.253 to outside:172.18.1.1 ID=3 seq=2 len=72

ICMP echo request from outside:172.18.1.1 to inside:172.30.0.253 ID=3 seq=3 len=72

ICMP echo reply from inside:172.30.0.253 to outside:172.18.1.1 ID=3 seq=3 len=72

Here is the show crypto ipsec sa:

ASA# sh cry ipsec sa

interface: outside

Crypto map tag: IPsec_RA, seq num: 10, local addr: 157.22.21.248

local ident (addr/mask/prot/port): (0.0.0.0/0.0.0.0/0/0)

remote ident (addr/mask/prot/port): (172.30.1.1/255.255.255.255/0/0)

current_peer: 76.199.70.100, username: vocera

dynamic allocated peer ip: 172.30.1.1

#pkts encaps: 212, #pkts encrypt: 212, #pkts digest: 212

#pkts decaps: 441, #pkts decrypt: 441, #pkts verify: 441

#pkts compressed: 0, #pkts decompressed: 0

#pkts not compressed: 212, #pkts comp failed: 0, #pkts decomp failed: 0

#pre-frag successes: 0, #pre-frag failures: 0, #fragments created: 0

#PMTUs sent: 0, #PMTUs rcvd: 0, #decapsulated frgs needing reassembly: 0

#send errors: 0, #recv errors: 0

local crypto endpt.: 157.22.21.248/4500, remote crypto endpt.: 76.199.70.100/50235

path mtu 1500, ipsec overhead 66, media mtu 1500

current outbound spi: CEE25F98

inbound esp sas:

spi: 0x47706B9D (1198549917)

transform: esp-3des esp-sha-hmac none

in use settings ={RA, Tunnel, NAT-T-Encaps, }

slot: 0, conn_id: 9, crypto-map: IPsec_RA

sa timing: remaining key lifetime (sec): 28173

IV size: 8 bytes

replay detection support: Y

outbound esp sas:

spi: 0xCEE25F98 (3470942104)

transform: esp-3des esp-sha-hmac none

in use settings ={RA, Tunnel, NAT-T-Encaps, }

slot: 0, conn_id: 9, crypto-map: IPsec_RA

sa timing: remaining key lifetime (sec): 28170

IV size: 8 bytes

replay detection support: Y

husycisco Sat, 12/13/2008 - 02:21

Hey Jorge!

So nice to hear from you m8 :) Hope you are doing well. I also had a look at question but couldnt draw the picture in mind mind so passed to other questions. Got back when Jian asked again, and drew the diagram on a paper :)

Actually the following line puzzles me " IPsec clients from branch office can not ping hosts in branch office but can ping "anywhere else""

Which clients located where are these "anywhere else"? Because being able to ping the remote sites from VPN client in this case requires an outside nat. I think nat-control is disbaled. I recommend you to refrain using PING to test connectivity, especially in networks have covering addresses like yours instead, enable remote desktop at remote end, then use "telnet remoteendip 3389" and see if you get a blank screen

Btw a little side note, temporarily disable windows firewall if enabled at remote end. Windows firewall exceptions do work only for same subnet, it has to be entered manually.

Output of following would also help

packet-tracer input outside tcp 1324 172.30.1.1 3389 172.30.0.253 detailed

jiangu Sat, 12/13/2008 - 09:24

Hi,

"anywhere else" means hosts in HQ inside network, IPsec software clients terminated on HQ ASA and ezvpn hosts terminated on HQ ASA.

Since IPsec RA clients terminated on branch ASA can ping "anywhere else", so it can not be a windows firewall issue, from the last post, debug icmp trace clearly showed that the echo reply packets from branch inside host (172.30.0.253) is being looped back from branch ASA's outside interface if the ping is initiated from branch ASA IPsec RA client.

CMP echo request from outside:172.30.1.1 to inside:172.30.0.253 ID=1 seq=80 len=32 <== OK

ICMP echo reply from inside:172.30.0.253 to outside:172.30.1.1 ID=1 seq=80 len=32 <== OK

ICMP echo reply from outside:172.30.0.253 to outside:172.30.1.1 ID=1 seq=80 len=32 <==???

ICMP echo reply from outside:172.30.0.253 to outside:172.30.1.1 ID=1 seq=80 len=32 <== ???

ICMP echo reply from outside:172.30.0.253 to outside:172.30.1.1 ID=1 seq=80 len=32

ICMP echo reply from outside:172.30.0.253 to outside:172.30.1.1 ID=1 seq=80 len=32

ICMP echo reply from outside:172.30.0.253 to outside:172.30.1.1 ID=1 seq=80 len=32

ICMP echo reply from outside:172.30.0.253 to outside:172.30.1.1 ID=1 seq=80 len=32

(the rest snipped...)

husycisco Sat, 12/13/2008 - 10:17

"Since IPsec RA clients terminated on branch ASA can ping "anywhere else", so it can not be a windows firewall issue,"

I meant the firewall in host you are pinging to in inside network...

Can you post the packet tracer output?

jiangu Sat, 12/13/2008 - 10:26

I can not VPN in right now, but hosts in "anywhere else" can ping 172.30.0.253, which is actually a Cisco router, not a windows host. I will post packet tracer later, if you still think that is going to helpful. What's you comment about echo reply constantly looping from outside to inside?

JORGE RODRIGUEZ Sat, 12/13/2008 - 11:45

Post packet tracer suggested by Huseyin , it will definately help

question on topology there is a route statement in asa.

route inside 172.30.0.0 255.255.240.0 172.30.0.252 1

what type of routes does 172.30.0.252 router have , defualt route pointing to asa inside interface? or is this statement suppose to be pointing to 172.30.0.253 instead of 252?

jiangu Sat, 12/13/2008 - 12:33

172.30.0.0 255.255.240.0 is default route pointing to a 3560 switch which terminates other 172.30.0.0/20 subnets. 172.30.0.253 is actually a CME router on stick. I will post packet tracer output when I get a chance. But I believe if I can resolve the mystery of echo reply looping inside ASA, the connectivity problem will go away.

husycisco Sat, 12/13/2008 - 12:44

You are right that fixing that loop will be the solution.

What happens when you issue management-access inside then ping 172.30.0.254 from vpn client?

Is VPN clients local area connection in a subnet that is covered by 172.16.0.0/20 ?

Try adding this,

policy-map global_policy

class inspection default

inspect icmp

jiangu Sat, 12/13/2008 - 13:19

Ping 172.30.0.254 has the same results, echo reply is looping inside ASA ...

When inspect icmp is added to global policy, client does not receive echo reply.

Here is the packet tracer output you requested:

ASA# packet-tracer input outside tcp 172.30.1.1 1234 172.30.0.253 22 $

Phase: 1

Type: CAPTURE

Subtype:

Result: ALLOW

Config:

Additional Information:

Forward Flow based lookup yields rule:

in id=0x39f3a18, priority=12, domain=capture, deny=false

hits=12468145, user_data=0x41479d8, cs_id=0x0, l3_type=0x0

src mac=0000.0000.0000, mask=0000.0000.0000

dst mac=0000.0000.0000, mask=0000.0000.0000

Phase: 2

Type: ACCESS-LIST

Subtype:

Result: ALLOW

Config:

Implicit Rule

Additional Information:

Forward Flow based lookup yields rule:

in id=0x39c0740, priority=1, domain=permit, deny=false

hits=14381155, user_data=0x0, cs_id=0x0, l3_type=0x8

src mac=0000.0000.0000, mask=0000.0000.0000

dst mac=0000.0000.0000, mask=0000.0000.0000

Phase: 3

Type: FLOW-LOOKUP

Subtype:

Result: ALLOW

Config:

Additional Information:

Found no matching flow, creating a new flow

Phase: 4

Type: ROUTE-LOOKUP

Subtype: input

Result: ALLOW

Config:

Additional Information:

in 172.30.0.0 255.255.255.0 inside

Phase: 5

Type: ACCESS-LIST

Subtype:

Result: DROP

Config:

Implicit Rule

Additional Information:

Forward Flow based lookup yields rule:

in id=0x39c0980, priority=0, domain=permit, deny=true

hits=9098, user_data=0x9, cs_id=0x0, flags=0x1000, protocol=0

src ip=0.0.0.0, mask=0.0.0.0, port=0

dst ip=0.0.0.0, mask=0.0.0.0, port=0, dscp=0x0

Result:

input-interface: outside

input-status: up

input-line-status: up

output-interface: inside

output-status: up

output-line-status: up

Action: drop

Drop-reason: (acl-drop) Flow is denied by configured rule

husycisco Sat, 12/13/2008 - 15:17

Can you temporarily put a permit ip any any in outside interface before running packet-trace? Btw sometimes reloading the device resolves weird issues like this one. What do you see in capture if you ping VPN client from router?

jiangu Sat, 12/13/2008 - 15:28

permit ip any any did not help, I didn't think it would help because the traffic is coming from IPsec tunnel... ping from inside has the same results, aka, ping packets loop inside ASA. I reloaded several times, still no go, this is really frustrating... I will try to upgrade to 8.04 next week.

Correct Answer
husycisco Sat, 12/13/2008 - 16:35

"permit ip any any did not help, I didn't think it would help because the traffic is coming from IPsec tunnel"

My intend was "run packet-tracer again after permitting any to any at outside interface"

Assuming that you have logging enabled, what syslog messages show up when you try to telnet into that router?

Can you create a tunnel-group test and assign a pool which is not covered by any of your inside networks, then create exempt nat and split tunnel acls accordingly? Are results the same in test group with new pool?

Also try

no crypto isakmp nat-traversal 10

crypto isakmp nat-traversal 30

jiangu Sat, 12/13/2008 - 17:08

Changing RA pool to 192.168.100.0/24 resolved the problem, what does this mean?

jiangu Sat, 12/13/2008 - 19:26

Hi,

This is extremely helpful, I reconfigured all ACLs (nat 0, split-tunnl and ACL for L2L) to be very specific, instead of the loose ones I originally configured, and the problem is resolved. Looks like ASA is not doing longest-match for destination when looking up security policy database.

husycisco Sun, 12/14/2008 - 02:53

Hello Jian,

Nice to hear that problem is resolved. I usually recommend people to use a VPN subnet which is not covered by any other subnets, that causes issues, subordinal routing and hard troubleshooting, but in your issue, you had reverse-route, which should have created a static route in routing table and should be preferred since it has /32 prefix. I am still curious about the packet-tracer output of your non-workng config. I will load your config in my lab when I have time and work on it, I couldnt find a bug close to yours in bug database, we might have discovered a new bug.

JORGE RODRIGUEZ Sun, 12/14/2008 - 06:13

Im curious as well to find out why these two nets 172.30.0.0/20 and 172.30.1.0/24 did not route properly.. this is code 7.2(4) and as you said it could very well be a newly discovered bug.. definately grants a lab test .

Points well diserved Huseying !

Always good to see your around.

Rgds

Jorge

jiangu Sun, 12/14/2008 - 09:41

I don't think reverse-route makes a difference though in my case, as I am not redistributing the static route out to other network devices. The reason I choose loose ACLs is for convenience, because this way whenever a new subnet is added, I don't have to touch network devices, I assume ASA will always to longest match, that has been working fine,

Thank both of you very much for your time.

Actions

This Discussion