L2L VPN Issue - one subnet not reachable

[darragh long]

Hi Folks,

I have a strange issue with a new VPN connection and would appreciate any help.

I have a pair of Cisco asa 5540s configured as a failover pair (code version 8.2(5)).   

I have recently added 2 new L2L VPNs - both these VPNs are sourced from the same interface on my ASA (called isp), and both are to the same customer, but they terminate on different firewalls on the cusomter end, and encrypt traffic from different customer subnets.    There's a basic network diagram attached.

VPN 1 - is for traffic from the customer subnet 10.2.1.0/24.    Devices in this subnet should be able to access 2 subnets on my network - DMZ 211 (192.168.211.0./24) and DMZ 144 (192.168.144.0/24).    This VPN works correctly.

VPN 2 - is for traffic from the customer subnet 192.168.1.0/24.    Devices in  this subnet should be able to access the same 2 subnets on my network - DMZ 211  (192.168.211.0./24) and DMZ 144 (192.168.144.0/24).    This VPN is not working correctly - the customer can access DMZ 144, but not DMZ 211.

There are isakmp and ipsec SAs for both VPNs.    I've noticed that the packets encaps/decaps counter does not increment when the customer sends test traffic to DMZ 211.  This counter does increment when they send test traffic to DMZ144.   I can also see traffic sent to DMZ 144 from the customer subnet 192.168.1.0/24 in packet captures on the DMZ 144 interface of the ASA.   I cannot see similar traffic in captures on the DMZ211 interface (although I can see traffic sent to DMZ211 if it is sourced from 10.2.1.0/24 - ie when it uses VPN1)

Nat exemption is configured for both 192.168.1.0/24 and 10.2.1.0/24.

There is a route to both customer subnets via the same next hop.

There is nothing in the logs toindicate that traffic from 192.168.1.0/24 is being dropped

I suspect that this may be an issue on the customer end, but I'd like to be able to prove that.   Specifically, I would really like to be able to capture traffic destined to DMZ 211 on the isp interface of the firewall after it has been decrypted - I don't know if this can be done however, and I haven'treally found a good way to prove or disprove that VPN traffic from 192.168.1.0/24 to DMZ211 is arriving at the isp interface of my ASA, and to show what's happening to that traffic after it arrives.

Here is the relevant vpn configuration:

crypto map MY_CRYPTO_MAP 90 match address VPN_2

crypto map MY_CRYPTO_MAP 90 set peer 217.154.147.221

crypto map MY_CRYPTO_MAP 90 set transform-set 3dessha

crypto map MY_CRYPTO_MAP 90 set security-association lifetime seconds 86400

crypto map MY_CRYPTO_MAP 100 match address VPN_1

crypto map MY_CRYPTO_MAP 100 set peer 193.108.169.48

crypto map MY_CRYPTO_MAP 100 set transform-set 3dessha

crypto map MY_CRYPTO_MAP 100 set security-association lifetime seconds 86400

crypto map MY_CRYPTO_MAP interface isp

ASA# sh access-list VPN_2

access-list VPN_2; 6 elements; name hash: 0xa902d2f4

access-list VPN_2 line 1 extended permit ip object-group VPN_2_NETS 192.168.1.0 255.255.255.0 0x56c7fb8f

  access-list VPN_2 line 1 extended permit ip 192.168.144.0 255.255.255.0 192.168.1.0 255.255.255.0 (hitcnt=45) 0x93b6dc21

  access-list VPN_2 line 1 extended permit ip 192.168.211.0 255.255.255.0 192.168.1.0 255.255.255.0 (hitcnt=6) 0x0abf7bb9

  access-list VPN_2 line 1 extended permit ip host 192.168.146.29 192.168.1.0 255.255.255.0 (hitcnt=8) 0xcc48a56e

ASA# sh access-list VPN_1

access-list VPN_1; 3 elements; name hash: 0x30168cce

access-list VPN_1 line 1 extended permit ip 192.168.144.0 255.255.252.0 10.2.1.0 255.255.255.0 (hitcnt=6) 0x61759554

access-list VPN_1 line 2 extended permit ip 192.168.211.0 255.255.255.0 10.2.1.0 255.255.255.0 (hitcnt=3) 0xa602c97c

access-list VPN_1 line 3 extended permit ip host 192.168.146.29 10.2.1.0 255.255.255.0 (hitcnt=0) 0x7b9f32e3

nat (dmz144) 0 access-list nonatdmz144

nat (dmz211) 0 access-list nonatdmz211

ASA# sh access-list nonatdmz144

access-list nonatdmz144; 5 elements; name hash: 0xbf28538e

access-list nonatdmz144 line 1 extended permit ip 192.168.144.0 255.255.255.0 192.168.0.0 255.255.0.0 (hitcnt=0) 0x20121683

access-list nonatdmz144 line 2 extended permit ip 192.168.144.0 255.255.255.0 172.28.2.0 255.255.254.0 (hitcnt=0) 0xbc8ab4f1

access-list nonatdmz144 line 3 extended permit ip 192.168.144.0 255.255.255.0 194.97.141.160 255.255.255.224 (hitcnt=0) 0xce869e1e

access-list nonatdmz144 line 4 extended permit ip 192.168.144.0 255.255.255.0 172.30.0.0 255.255.240.0 (hitcnt=0) 0xd3ec5035

access-list nonatdmz144 line 5 extended permit ip 192.168.144.0 255.255.255.0 10.2.1.0 255.255.255.0 (hitcnt=0) 0x4c9cc781

ASA# sh access-list nonatdmz211 | in 192.168\.1\.

access-list nonatdmz1 line 3 extended permit ip 192.168.211.0 255.255.255.0 192.168.1.0 255.255.255.0 (hitcnt=0) 0x2bbfcfdd

ASA# sh access-list nonatdmz211 | in 10.2.1.

access-list nonatdmz1 line 4 extended permit ip 192.168.211.0 255.255.255.0 10.2.1.0 255.255.255.0 (hitcnt=0) 0x8a836d91

route isp 192.168.1.0 255.255.255.0 137.191.234.33 1

route isp 10.2.1.0 255.255.255.0 137.191.234.33 1

Thanks in advance to anyone who gets this far!

[Richard Burts]

I have read through the post and am a bit confused. You show us what appears to be all of one access list

sh access-list nonatdmz144

and you tell us that this dmz is working. I see an exemption for 10.2.1.0 but do not see an exemption for 192.168.1.0.

And in the nat exemption for the dmz that you say is not working I see exemption for both remote subnets.

Perhaps you can clarify?

HTH

Rick

[darragh long]

Hi Rick,

There isn't a specific entry in ACL nonatdmz144 for 192.168.1.0/24, but the first entry in that ACL covers the 192.168.0.0/16 subnet, so I would expect 192.168.1.0/24 to be included with that?

It's correct that traffic to dmz144 is working (from both customer subnets), and that there is a nat exemption for both customer subnets when accessing dmz211.

Thanks,

Darragh

[Richard Burts]

Darragh

Thank you for the clarification. I had seen the 192.168.0.0 in the access list but missed the detail that it was /16. So yes that should certainly cover 192.168.1.0.

I do notice that the vpn access list for vpn 2 has a hit count of 6 for the traffic from the remote subnet

  access-list VPN_2 line 1 extended permit ip 192.168.211.0 255.255.255.0 192.168.1.0 255.255.255.0 (hitcnt=6) 0x0abf7bb9

which suggests that the traffic is getting to your ASA. Is there any possibility that there is something in dmz211 that is doing any traffic filtering, or hosts within that dmz that are selective about what they process?

HTH

Rick

[darragh long]

Hi Rick,

I think those hits were caused by the packet-tracer command.    

I just cleared the counters on access-list VPN_2 and checked the status of the VPN - there was no ISAKMP SA for VPN2 (I presume it timed out due to inactivity).    I called the customer to ask him to test, and he told me that he had been running a continuous ping to a host in DMZ211.     This didn't cause the VPN to come up (and naturally there were no hits on the crypto ACL).     I asked him to send traffic to a host in DMZ144 - this traffic incremented the hit counters on the crypto ACL, and caused the VPN to come up.

I checked the logs, and couldn't see any reference to traffic from the 192.168.1.0 subnet.

It looks to me like traffic from 192.168.1.0/24 destined fro 192.168.211.0/24 is not being routed to my firewall, but I'd really like to use a packet capture on my isp interface to prove this is the case - do you know if that's possible?

Thanks,

Darragh

[Richard Burts · Accepted solution]

Darragh

Clearing the counters was a good idea. If the counter is not incrementing and if ping from the remote side is not causing the VPN to come up it certainly confirms that something is not working right.

It might be interesting to wait till the SAs time out and go inactive and then test again with the ping from the remote subnet that is not working. Turn on debug for ISAKMP and see if there is any attempt to negotiate. Especially if you do not receive any attempt to initiate ISAKMP from then then that would be one way to show that there is a problem on the remote side.

Certainly the ASA does have the ability to do packet capture. I have used that capability and it can be quite helpful. I have not tried to do a capture on the outside interface for incoming VPN traffic and so am not sure whether you would be capturing the encrypted packet or the de-encrypted packet. You can configure an access list to identify traffic to capture and I guess that you could write an access list that included both the peer addresses as source and destination to capture the encrypted traffic and entries that were the un-encrypted source and destination subnets to capture traffic after de-encryption.

HTH

Rick

[darragh long]

Hi Rick,

Thanks for this.  I cleared down the SA, turned on ISAKMP debugging,and put two captures on the isp interface (one checking for any traffic from the peer, and one checking for any isakmp traffic).     When the customer tried to access DMZ211 I didn't see anything in the captures, or any debug output.    When he tried to access DMZ144 the captures and the debug both recorded it.    This was enough to prove that the issue was on his firewall, and he found an old encryption rule that overlapped with DMZ211, which was causing traffic to be routed incorrectly.   Thanks again for your help!

[Richard Burts]

I am glad that you got the problem resolved and that my suggestions were helpful. Thank you for posting back to the forum to tell us that it is solved and what the problem was. And thank you for using the rating system to mark this question as answered.

HTH

Rick