Odd connectivity problem - spoke and hub

atn2detail · ‎01-24-2014

Hi all.

I have a couple ASA 5505 in use at two separate locations. They are configured in Spoke/Hub; the two are linked with site to site vpn, and remote access vpn to main site will also allow traffic to traverse to the second site.

Everything was working great. Then main site needed to move to a new location, which also resulted in new ISP and new public IPs.

Both still get internet access, the remote access vpn still works fine. The site to site VPN is acting a little strangely. It was working just fine for a week. Yesterday, the site to site VPN stopped functioning properly. I cannot ping or navigate to the remote site anymore. If i use packet-tracer (packet-tracer input inside icmp 192.168.1.50 1 1 192.168.2.50) it will drop encrypted traffic due to ACL. However, when I try the same command immediately after (or any other ip to any other ip) the vpn will encrypt and allow flow-creation. Alas, I still cannot ping remote hosts.

I have verified and double checked that there is no firewall/anti-virus interference on the remote host that I am attempting to connect to. I have also checked and double checked the following:

-Remote site internet connectivity is fine.

-Remote site IP addresses have not changed.

-Strange behavior is present from both firewalls when trying to use packet-tracer

-Unable to ping from either direction in the site to site VPN. Remote access VPN can still ping the main site local hosts.

-Crypto map settings appear correct.

I would gladly post configs, but at this very moment I do not have access to them. Any help/suggestions would be appreciated.

Jouni Forss · ‎01-25-2014

Hi,

The "packet-tracer" command you used doesnt really correspond to the typical ICMP that a host would send.

You would use

packet-tracer input inside icmp 192.168.1.50 8 0 192.168.2.50

Also, if the L2L VPN connection is down and you attempt to use "packet-tracer" the result is always a VPN Phase DROP. This is because the first "packet-tracer" initiates the VPN negotiation and usually when you issue the same command again the VPN negotiation has already finished.

Can you confirm that the traffic is sent succesfully encrypted/encapsulated through the L2L VPN when you attempt the connections or send ICMP Echos?

You can naturally use the command

show crypto ipsec sa

or

show crypto ipsec sa peer

And first confirm that traffic is getting to the VPN connection.

Next you should probably confirm whats seen on the remote site. Does it decrypt/decapsulate traffic?

Naturally as one site moved I would confirm that no configurations were removed by accident.

You could go as far as capturing traffic on the ASAs and/or the hosts to confirm where the flow of traffic stops.

On some occasions this kind of troubleshooting might simply end in a situation where everything is configured correctly and there is no clear reason for the problem. In this case it might be some bug.

Sometimes the solution might simply have been to reconfigure the VPN connection from scratch. I have seen this work for some here in CSC also.

But I would suggest first following where the traffic flow stops with captures, logs and output of the different counters on the ASAs.

- Jouni

atn2detail · ‎01-25-2014

I used the version you recommended from both ends of the L2L tunnel. The result was the same; both allow traffic all the way through flow-creation.

I then verified security associations via show crypto ipsec sa. Just to be thorough, i used both methods you stated from both ends of the L2L tunnel. Both show the SA.

I'm still just getting my feet wet in Cisco, where can I see if the decryption is actually happening?

As far as configurations removed by accident, I keep a change log of every change command I issue to the ASA. Wouldn't the configuration removal have resulted in a downtime immediately rather than a week after the fact?

Thanks so much for your assistance.

Jouni Forss · ‎01-25-2014

Hi,

True, the configuration changes should have shown earlier. Was too much caught up listing things I didn't really think that one through

If I were to presume that the L2L VPN doesnt work at all at the moment and if there is very few attempts to pass traffic through it you could simply look at the counters of the command

show crypto ipsec sa

or

show crypto ipsec sa peer

You should see counters for SAs / network pairs on the L2L VPN connection and how much traffic has been encrypted/encapsulated (sent) and how much has been decrypted/decapsulated (received). This would be the fast way to determine if you are seeing the traffic on both ASA units.

Naturally you could use ASA logs to confirm the connection is built on the originating ASA and see if the same connection can be seen built on the remote ASA. If that is the case then you could naturally start confirming that the remote end sees some return traffic for this connection that has been built on the ASA (ASA builds a connection as long as it allows the first packet of the connection through itself)

The ASAs could be configured to capture the traffic you want and you could then very clearly confirm where the traffic stops.

Just to give you an example of a capture configuration on the ASA

access-list VPN-CAPTURE permit ip

capture VPN-CAPTURE type raw-data access-list VPN-CAPTURE interface inside buffer 5000000 circular buffer

You can naturally change the capture and ACL names and modify the ACL to include only certain local/remote host or even go as far as defining ports (though then have to make sure that the ACL really matches both directions of traffic). Also your local interface might no be named the default "inside".

You can then attempt connections and issue the following command to determine if anything has been captured

show capture

If traffic was captured you could then use the following command to show the output on the CLI

show capture VPN-CAPTURE

I would suggest though that you would attempt sending the capture with TFTP to some host so you can open it up with Wireshark for example

copy /pcap capture:VPN-CAPTURE tftp://x.x.x.x/VPN-CAPTURE.pcap

You can then remove the capture and its contents from the ASA with the command

no capture VPN-CAPTURE

The ACL you will have to remove separately.

But as I said, checking the logs while making connection attempts and checking the VPN counters might be easier things to start with to get a picture where the traffic stops and the capture could then be used at either device to get a clealer look at the situation.

As you mentioned already, using "packet-tracer" command that matches the L2L VPN configuration already provides a output that indicates that the VPN configuration is matched and the VPN negotiation goes through.

As for possible configuration changes that might have happend on the ASA I will mention this even though it doesnt match your problem as it started well after the ISP change.

In certain situation where people use the ASDM and its Wizard to configure a VPN they might change the setting that enables the VPN traffic to Bypass the interface ACLs. This configuration change might essentially cause that either end ASA might start blocking the incoming traffic from VPN. I mean the traffic that is arriving to one of the ASA firewalls and being decrypted/decapsulated.

The command on the CLI is

sysopt connection permit-vpn

This is the default setting on an ASA and doesnt show up in the basic CLI configuration

no sysopt connection permit-vpn

This setting shows up in the CLI configuration and also means that the ACL of the interface that builds the VPN connection now controls traffic coming even from VPN connection.

Hope this helps

- Jouni

atn2detail · ‎01-25-2014

So interestingly, while the packet tracer works.... no packets are going out when I ping. The captures are 0 bytes after multiple ping attempts, and encryption/decryption all read 0 on the security associations. However when I run packet-tracer again with the icmp 8 0 packet, it goes through and generates 62 bytes.

atn2detail · ‎01-25-2014

More and more I suspect the firewall is not the problem, especially since no configurations were changed. (Only myself and my partner have access to the configs, and he was busy with another location when the outage occured). Could anything change on the ISP end that would affect this? The main site (the one that changed locations) also has a remote-access VPN configured. That VPN is completely functional and fine.

atn2detail · ‎01-27-2014

I have just noticed now that packet-tracer fails on VPN (subtype: encrypt) the first time I run it. However, if I run the packet-tracer again the vpn encryption does not fail.

Thoughts?

Jouni Forss · ‎01-28-2014

Hi,

Mentioned about that earlier with the "packet-tracer" command. Its normal behaviour if the VPN connection is down during the first time you issue the command

packet-tracer input inside icmp 192.168.1.50 8 0 192.168.2.50

Also, if the L2L VPN connection is down and you attempt to use "packet-tracer" the result is always a VPN Phase DROP. This is because the first "packet-tracer" initiates the VPN negotiation and usually when you issue the same command again the VPN negotiation has already finished.

- Jouni

atn2detail · ‎01-29-2014

I'm sorry, was a little sleep deprived and must have forgotten you already said something about it. So what must be happening then is that the firewalls are indeed connecting via VPN, and the problem lies elsewhere?

atn2detail · ‎01-30-2014

Got back in town and was able to get running configs. Two things here confuse me. Maybe I have a configuration wrong and my brain is just not letting me see the error.

1) I set up a site-to-site VPN from a different firewall (5505 but different software version) and it exhibited the exact same behavior. I could use Packet-Tracer, but not actually ping anything or initiate network traffic across it.

2) I configured a remote access VPN for FW2 and it works completely fine. I am able to ping the host through the remote-access VPN where I still cannot via site-to-site VPN.

FW1 public ip information has been replaced with 1.1.1.1

FW2 public ip information has been replaced with 2.2.2.2

Thanks for the help.

FW1

____________________________________________

ASA Version 8.2(5)

!

hostname fw1

enable password /r2oDuwuK7ckZKJB encrypted

passwd lcALn1AgoxqCj3uZ encrypted

names

!

interface Ethernet0/0

switchport access vlan 2

!

interface Ethernet0/1

!

interface Ethernet0/2

!

interface Ethernet0/3

!

interface Ethernet0/4

!

interface Ethernet0/5

!

interface Ethernet0/6

!

interface Ethernet0/7

!

interface Vlan1

nameif inside

security-level 100

ip address 192.168.1.1 255.255.255.0

!

interface Vlan2

nameif outside

security-level 0

ip address 1.1.1.1 255.255.255.128

!

ftp mode passive

dns domain-lookup outside

dns server-group defaultdns

name-server 1.1.1.1

same-security-traffic permit intra-interface

object-group network Exchange

network-object host 192.168.1.102

access-list outside_access_in extended permit tcp any interface outside eq 3389

access-list outside_access_in extended permit tcp any interface outside eq ldap

access-list outside_access_in extended permit tcp any interface outside eq 379

access-list outside_access_in extended permit tcp any interface outside eq 390

access-list outside_access_in extended permit tcp any interface outside eq 3268

access-list outside_access_in extended permit tcp any interface outside eq ldaps

access-list outside_access_in extended permit tcp any interface outside eq 3269

access-list outside_access_in extended permit tcp any interface outside eq imap4

access-list outside_access_in extended permit tcp any interface outside eq 993

access-list outside_access_in extended permit tcp any interface outside eq pop3

access-list outside_access_in extended permit tcp any interface outside eq 563

access-list outside_access_in extended permit tcp any interface outside eq www

access-list outside_access_in extended permit tcp any interface outside eq https

access-list outside_access_in extended permit tcp any interface outside eq smtp

access-list outside_access_in extended permit tcp any interface outside eq 465

access-list outside_access_in extended permit tcp any interface outside eq 691

access-list outside_access_in extended permit tcp any interface outside eq 102

access-list outside_access_in extended permit tcp any interface outside eq 135

access-list outside_access_in extended permit tcp any interface outside eq 522

access-list outside_access_in extended permit tcp any interface outside eq domain

access-list outside_access_in extended permit tcp any interface outside eq 717

access-list outside_access_in extended permit tcp any interface outside eq 2525

access-list outside_access_in extended permit tcp any interface outside eq 587

access-list nonat remark ACL for Nat Bypass

access-list nonat extended permit ip 192.168.1.0 255.255.255.0 10.11.12.0 255.255.255.0

access-list nonat extended permit ip 192.168.1.0 255.255.255.0 192.168.2.0 255.255.255.0

access-list vpn-SplitTunnel remark ACL for VPN Split Tunnel

access-list vpn_SplitTunnel standard permit 192.168.1.0 255.255.255.0

access-list vpn_SplitTunnel standard permit 192.168.2.0 255.255.255.0

access-list nsfw2_LAN extended permit ip 192.168.1.0 255.255.255.0 192.168.2.0 255.255.255.0

access-list nsfw2_LAN extended permit ip 10.11.12.0 255.255.255.0 192.168.2.0 255.255.255.0

pager lines 24

mtu inside 1500

mtu outside 1500

ip local pool vpnpool 10.11.12.1-10.11.12.255

icmp unreachable rate-limit 1 burst-size 1

no asdm history enable

arp timeout 14400

global (outside) 10 interface

nat (inside) 0 access-list nonat

nat (inside) 10 192.168.1.0 255.255.255.0

static (inside,outside) tcp interface 3389 192.168.1.100 3389 netmask 255.255.255.255

static (inside,outside) tcp interface ldap 192.168.1.102 ldap netmask 255.255.255.255

static (inside,outside) tcp interface 379 192.168.1.102 379 netmask 255.255.255.255

static (inside,outside) tcp interface 390 192.168.1.102 390 netmask 255.255.255.255

static (inside,outside) tcp interface 3268 192.168.1.102 3268 netmask 255.255.255.255

static (inside,outside) tcp interface ldaps 192.168.1.102 ldaps netmask 255.255.255.255

static (inside,outside) tcp interface 3269 192.168.1.102 3269 netmask 255.255.255.255

static (inside,outside) tcp interface imap4 192.168.1.102 imap4 netmask 255.255.255.255

static (inside,outside) tcp interface 993 192.168.1.102 993 netmask 255.255.255.255

static (inside,outside) tcp interface pop3 192.168.1.102 pop3 netmask 255.255.255.255

static (inside,outside) tcp interface 563 192.168.1.102 563 netmask 255.255.255.255

static (inside,outside) tcp interface www 192.168.1.102 www netmask 255.255.255.255

static (inside,outside) tcp interface https 192.168.1.102 https netmask 255.255.255.255

static (inside,outside) tcp interface smtp 192.168.1.102 smtp netmask 255.255.255.255

static (inside,outside) tcp interface 465 192.168.1.102 465 netmask 255.255.255.255

static (inside,outside) tcp interface 691 192.168.1.102 691 netmask 255.255.255.255

static (inside,outside) tcp interface 102 192.168.1.102 102 netmask 255.255.255.255

static (inside,outside) tcp interface 135 192.168.1.102 135 netmask 255.255.255.255

static (inside,outside) tcp interface 522 192.168.1.102 522 netmask 255.255.255.255

static (inside,outside) tcp interface domain 192.168.1.102 domain netmask 255.255.255.255

static (inside,outside) tcp interface 717 192.168.1.102 717 netmask 255.255.255.255

static (inside,outside) tcp interface 2525 192.168.1.102 2525 netmask 255.255.255.255

static (inside,outside) tcp interface 587 192.168.1.102 587 netmask 255.255.255.255

access-group outside_access_in in interface outside

route outside 0.0.0.0 0.0.0.0 1.1.1.2 1

timeout xlate 3:00:00

timeout conn 1:00:00 half-closed 0:10:00 udp 0:02:00 icmp 0:00:02

timeout sunrpc 0:10:00 h323 0:05:00 h225 1:00:00 mgcp 0:05:00 mgcp-pat 0:05:00

timeout sip 0:30:00 sip_media 0:02:00 sip-invite 0:03:00 sip-disconnect 0:02:00

timeout sip-provisional-media 0:02:00 uauth 0:05:00 absolute

timeout tcp-proxy-reassembly 0:01:00

timeout floating-conn 0:00:00

dynamic-access-policy-record DfltAccessPolicy

aaa authentication http console LOCAL

aaa authentication ssh console LOCAL

http server enable

http 192.168.1.0 255.255.255.0 inside

no snmp-server location

no snmp-server contact

snmp-server enable traps snmp authentication linkup linkdown coldstart

crypto ipsec transform-set strong-des esp-3des esp-md5-hmac

crypto ipsec transform-set nsfw2 esp-aes-256 esp-sha-hmac

crypto ipsec security-association lifetime seconds 28800

crypto ipsec security-association lifetime kilobytes 4608000

crypto dynamic-map dynmap 30 set transform-set strong-des

crypto map ns 1 match address nsfw2_LAN

crypto map ns 1 set peer 2.2.2.2

crypto map ns 1 set transform-set nsfw2

crypto map ns 65535 ipsec-isakmp dynamic dynmap

crypto map ns interface outside

crypto isakmp identity address

crypto isakmp enable outside

crypto isakmp policy 1

authentication pre-share

encryption aes-256

hash sha

group 2

lifetime 86400

crypto isakmp policy 11

authentication pre-share

encryption 3des

hash md5

group 2

lifetime 86400

console timeout 0

management-access inside

dhcpd lease 3000

dhcpd option 3 ip 192.168.1.1

!

dhcpd address 192.168.1.120-192.168.1.250 inside

dhcpd enable inside

!

threat-detection basic-threat

threat-detection statistics access-list

no threat-detection statistics tcp-intercept

webvpn

group-policy ns internal

group-policy ns attributes

vpn-idle-timeout 120

vpn-tunnel-protocol IPSec

split-tunnel-policy tunnelspecified

split-tunnel-network-list value vpn_SplitTunnel

tunnel-group ns-VPN type remote-access

tunnel-group ns-VPN general-attributes

address-pool vpnpool

default-group-policy ns

tunnel-group ns-VPN ipsec-attributes

pre-shared-key *****

tunnel-group 2.2.2.2 type ipsec-l2l

tunnel-group 2.2.2.2 ipsec-attributes

pre-shared-key *****

!

class-map inspection_default

match default-inspection-traffic

!

policy-map type inspect dns preset_dns_map

parameters

message-length maximum client auto

message-length maximum 512

policy-map global_policy

class inspection_default

inspect dns preset_dns_map

inspect ftp

inspect h323 h225

inspect h323 ras

inspect ip-options

inspect netbios

inspect rsh

inspect rtsp

inspect skinny

inspect sqlnet

inspect sunrpc

inspect tftp

inspect sip

inspect xdmcp

inspect icmp

inspect icmp error

FW2

_______________________________

ASA Version 8.2(5)

!

hostname fw2

enable password rDlrx/ijZiwp44Mi encrypted

passwd Ft0mv6GdiaYo9Cge encrypted

names

!

interface Ethernet0/0

switchport access vlan 2

!

interface Ethernet0/1

!

interface Ethernet0/2

!

interface Ethernet0/3

!

interface Ethernet0/4

!

interface Ethernet0/5

!

interface Ethernet0/6

!

interface Ethernet0/7

!

interface Vlan1

nameif inside

security-level 100

ip address 192.168.2.1 255.255.255.0

!

interface Vlan2

nameif outside

security-level 0

ip address 2.2.2.2 255.255.255.248

!

ftp mode passive

access-list nsfw1_LAN extended permit ip 192.168.2.0 255.255.255.0 192.168.1.0 255.255.255.0

access-list nsfw1_LAN extended permit ip 192.168.2.0 255.255.255.0 10.11.12.0 255.255.255.0

access-list nonat remark ACL for NAT Bypass

access-list nonat extended permit ip 192.168.2.0 255.255.255.0 192.168.1.0 255.255.255.0

access-list nonat extended permit ip 192.168.2.0 255.255.255.0 10.11.12.0 255.255.255.0

access-list nonat extended permit ip 192.168.2.0 255.255.255.0 10.10.10.0 255.255.255.0

access-list vpn_SplitTunnel remark ACL for VPN Split Tunnel

access-list vpn_SplitTunnel standard permit 192.168.2.0 255.255.255.0

pager lines 24

mtu inside 1500

mtu outside 1500

ip local pool ravpnpool 10.10.10.1-10.10.10.10

icmp unreachable rate-limit 1 burst-size 1

no asdm history enable

arp timeout 14400

global (outside) 10 interface

nat (inside) 0 access-list nonat

nat (inside) 10 192.168.2.0 255.255.255.0

route outside 0.0.0.0 0.0.0.0 2.2.2.3 1

timeout xlate 3:00:00

timeout conn 1:00:00 half-closed 0:10:00 udp 0:02:00 icmp 0:00:02

timeout sunrpc 0:10:00 h323 0:05:00 h225 1:00:00 mgcp 0:05:00 mgcp-pat 0:05:00

timeout sip 0:30:00 sip_media 0:02:00 sip-invite 0:03:00 sip-disconnect 0:02:00

timeout sip-provisional-media 0:02:00 uauth 0:05:00 absolute

timeout tcp-proxy-reassembly 0:01:00

timeout floating-conn 0:00:00

dynamic-access-policy-record DfltAccessPolicy

aaa authentication ssh console LOCAL

no snmp-server location

no snmp-server contact

snmp-server enable traps snmp authentication linkup linkdown coldstart

crypto ipsec transform-set nsfw2 esp-aes-256 esp-sha-hmac

crypto ipsec transform-set ra-nsfw2 esp-3des esp-md5-hmac

crypto ipsec security-association lifetime seconds 28800

crypto ipsec security-association lifetime kilobytes 4608000

crypto dynamic-map dynmap 30 set transform-set ra-nsfw2

crypto map ns2 1 match address nsfw1_LAN

crypto map ns2 1 set peer 1.1.1.1

crypto map ns2 1 set transform-set nsfw2

crypto map ns2 65535 ipsec-isakmp dynamic dynmap

crypto map ns2 interface outside

crypto isakmp identity address

crypto isakmp enable outside

crypto isakmp policy 1

authentication pre-share

encryption aes-256

hash sha

group 2

lifetime 86400

crypto isakmp policy 12

authentication pre-share

encryption 3des

hash md5

group 2

lifetime 86400

console timeout 0

management-access inside

!

dhcpd address 192.168.2.60-192.168.2.200 inside

dhcpd enable inside

!

threat-detection basic-threat

threat-detection statistics access-list

no threat-detection statistics tcp-intercept

group-policy ns2 internal

group-policy ns2 attributes

vpn-idle-timeout 120

vpn-tunnel-protocol IPSec

split-tunnel-policy tunnelspecified

split-tunnel-network-list value vpn_SplitTunnel

tunnel-group 1.1.1.1 type ipsec-l2l

tunnel-group 1.1.1.1 ipsec-attributes

pre-shared-key *****

tunnel-group ph-VPN type remote-access

tunnel-group ph-VPN general-attributes

address-pool ravpnpool

default-group-policy ns2

tunnel-group ph-VPN ipsec-attributes

pre-shared-key *****

!

class-map inspection_default

match default-inspection-traffic

!

policy-map type inspect dns preset_dns_map

parameters

message-length maximum client auto

message-length maximum 512

policy-map global_policy

class inspection_default

inspect dns preset_dns_map

inspect ftp

inspect h323 h225

inspect h323 ras

inspect ip-options

inspect netbios

inspect rsh

inspect rtsp

inspect skinny

inspect esmtp

inspect sqlnet

inspect sunrpc

inspect tftp

inspect sip

inspect xdmcp

inspect icmp

inspect icmp error

atn2detail · ‎01-31-2014

Bump, looking for help still.

atn2detail · ‎02-03-2014

Bump again please. We have a workaround in place but I would like to have this tunnel functioning as it should.