06-15-2010 10:22 AM - edited 03-11-2019 10:59 AM
I'm having a problem with one of two ASA 5505s dropping outbound connections. Effectively the output traffic on the outside interface goes to zero.
The ASAs are configured for a single IPSec tunnel and routing/firewall on either end.
Intermittently one ASA will drop outbound connections. Sometimes it will come back on its own, other times I have to either power cycle the device or log into ASDM and do a system reload. The syslog output doesn't show much other than it dropped the tunnel
It doesn't seem to be system load related since it will do this basically any time. Late at night when the office is closed or during the day.
This config has worked for over 9 months and just recently it started dropping outbound
I'd really appreciate any suggestions.
Result of the command: "show running"
: Saved
:
ASA Version 8.2(1)
!
hostname ciscoasa
domain-name default.domain.invalid
enable password Sl7iFbMSQUs45M7. encrypted
passwd Sl7iFbMSQUs45M7. encrypted
names
dns-guard
!
interface Vlan1
nameif inside
security-level 100
ip address 192.168.1.1 255.255.255.0
!
interface Vlan2
nameif outside
security-level 0
ip address 24.68.xxx.xxx 255.255.252.0
!
interface Ethernet0/0
switchport access vlan 2
!
interface Ethernet0/1
!
interface Ethernet0/2
!
interface Ethernet0/3
!
interface Ethernet0/4
!
interface Ethernet0/5
!
interface Ethernet0/6
!
interface Ethernet0/7
!
boot system disk0:/asa821-k8.bin
boot system disk0:/asa724-k8.bin
ftp mode passive
clock timezone PST -8
clock summer-time PDT recurring
dns domain-lookup inside
dns domain-lookup outside
dns server-group DefaultDNS
name-server 64.59.160.13
name-server 64.59.160.15
domain-name default.domain.invalid
access-list inside_nat0_outbound extended permit ip 192.168.1.0 255.255.255.0 192.168.2.0 255.255.255.0
access-list outside_access_in extended permit tcp any interface outside eq pptp
access-list outside_cryptomap extended permit ip 192.168.1.0 255.255.255.0 192.168.2.0 255.255.255.0
access-list Internal_ICMP extended permit icmp 192.168.2.0 255.255.255.0 192.168.1.0 255.255.255.0
pager lines 24
logging enable
logging timestamp
logging monitor warnings
logging trap debugging
logging asdm debugging
logging recipient-address
logging host inside 192.168.1.50
logging debug-trace
mtu inside 1500
mtu outside 1500
ip verify reverse-path interface outside
icmp unreachable rate-limit 1 burst-size 1
asdm image disk0:/asdm-621.bin
asdm history enable
arp timeout 14400
global (outside) 1 interface
nat (inside) 0 access-list inside_nat0_outbound
nat (inside) 1 0.0.0.0 0.0.0.0
static (inside,outside) tcp interface pptp 192.168.1.50 pptp netmask 255.255.255.255
access-group outside_access_in in interface outside
route outside 0.0.0.0 0.0.0.0 24.68.238.1 1
timeout xlate 3:00:00
timeout conn 1:00:00 half-closed 0:10:00 udp 0:02:00 icmp 0:00:02
timeout sunrpc 0:10:00 h323 0:05:00 h225 1:00:00 mgcp 0:05:00 mgcp-pat 0:05:00
timeout sip 0:30:00 sip_media 0:02:00 sip-invite 0:03:00 sip-disconnect 0:02:00
timeout sip-provisional-media 0:02:00 uauth 0:05:00 absolute
timeout tcp-proxy-reassembly 0:01:00
dynamic-access-policy-record DfltAccessPolicy
http server enable
http 192.168.1.0 255.255.255.0 inside
http 96.50.xxx.xxx 255.255.255.255 outside
no snmp-server location
no snmp-server contact
snmp-server enable traps snmp authentication linkup linkdown coldstart
crypto ipsec transform-set ESP-AES-256-MD5 esp-aes-256 esp-md5-hmac
crypto ipsec transform-set ESP-DES-SHA esp-des esp-sha-hmac
crypto ipsec transform-set ESP-DES-MD5 esp-des esp-md5-hmac
crypto ipsec transform-set ESP-AES-192-MD5 esp-aes-192 esp-md5-hmac
crypto ipsec transform-set ESP-3DES-MD5 esp-3des esp-md5-hmac
crypto ipsec transform-set ESP-AES-256-SHA esp-aes-256 esp-sha-hmac
crypto ipsec transform-set ESP-AES-128-SHA esp-aes esp-sha-hmac
crypto ipsec transform-set ESP-AES-192-SHA esp-aes-192 esp-sha-hmac
crypto ipsec transform-set ESP-AES-128-MD5 esp-aes esp-md5-hmac
crypto ipsec transform-set ESP-3DES-SHA esp-3des esp-sha-hmac
crypto ipsec security-association lifetime seconds 28800
crypto ipsec security-association lifetime kilobytes 4608000
crypto map outside_map1 2 match address outside_cryptomap
crypto map outside_map1 2 set pfs group1
crypto map outside_map1 2 set peer 96.50.xxx.xxx
crypto map outside_map1 2 set transform-set ESP-AES-128-SHA ESP-AES-128-MD5 ESP-AES-192-SHA ESP-AES-192-MD5 ESP-AES-256-SHA ESP-AES-256-MD5 ESP-3DES-SHA ESP-3DES-MD5 ESP-DES-SHA ESP-DES-MD5
crypto map outside_map1 2 set reverse-route
crypto map outside_map1 interface outside
crypto isakmp enable inside
crypto isakmp enable outside
crypto isakmp policy 10
authentication pre-share
encryption 3des
hash sha
group 2
lifetime 86400
telnet timeout 5
ssh timeout 5
console timeout 0
dhcpd dns 64.59.160.13 64.59.160.15
dhcpd auto_config outside
!
dhcpd address 192.168.1.2-192.168.1.33 inside
!
threat-detection basic-threat
threat-detection scanning-threat shun
threat-detection statistics
threat-detection statistics tcp-intercept rate-interval 30 burst-rate 400 average-rate 200
webvpn
group-policy DfltGrpPolicy attributes
vpn-tunnel-protocol IPSec l2tp-ipsec
tunnel-group 96.50.xxx.xxx type ipsec-l2l
tunnel-group 96.50.xxx.xxx ipsec-attributes
pre-shared-key *
!
class-map inspection_default
match default-inspection-traffic
!
!
policy-map type inspect dns preset_dns_map
parameters
message-length maximum 512
policy-map global_policy
class inspection_default
inspect dns preset_dns_map
inspect ftp
inspect h323 h225
inspect h323 ras
inspect rsh
inspect rtsp
inspect esmtp
inspect sqlnet
inspect skinny
inspect sunrpc
inspect xdmcp
inspect sip
inspect netbios
inspect tftp
inspect pptp
!
service-policy global_policy global
prompt hostname context
Cryptochecksum:6d3be823caf96c4e5d264e440d64c609
: end
Solved! Go to Solution.
07-12-2010 08:09 AM
Hi Jason,
OK, this tells us a couple of things:
1) The problem is not caused by the MAC of the Gateway IP changing (as would be the case if there was a duplicate IP for the GW IP)
2) Since the ASA has the correct ARP entry, and is forwarding the packets to the GW, then the problem resides with the GW
3) We know that clearing the ARP table on the ASA resolves the issue, so what does that do?
Well, when we clear the ARP table, and then ping, the ASA will send out an ARP request for the Destination IP, to resolve the MAC. That suceeds, as the ASA is populating it's ARP table again with the GW IP and MAC. Also, this then allows the GW to respond.
Therefore, I would have to speculate that the GW has an incorrect MAC entry for the ASA, and sending the ARP request from the ASA is updating the gateway's entry for the ASAs IP.
One thing that may allow us to detect this is to:
a) Configure a capture on the outside interface, capturing only ARP traffic
b) Set the capture to be a circular-buffer
c) Set the capture size to be like 5 MB
d) When the problem happens, pull the capture off the ASA
Example:
capture out interface outside ethernet-type arp buffer 5000000 circular-buffer
With this, we can look to see if we see another device using your IP at the time the problem occurs.
Sincerely,
David.
06-18-2010 11:42 AM
Hi Jason,
When you have the outbound drops - it sounds like no traffic egresses the outside interface? Or is it only tunneled traffic?
1) When you connect via ASDM, are you connecting to the outside interface, or the inside?
2) Access the ASA when you have the problem, and verify you can ping the adjacent devices out both interfaces. If that works, then try initiating traffic through the ASA - and record the "show conn" output as well as the syslogs.
3) If the tunnel is down, attempt to send traffic across the tunnel, and see if the tunnel comes up. Again, capture the 'show conn' and syslog outputs.
Thanks,
David.
06-22-2010 06:10 PM
Hi David,
Thanks for your reply.
The router is configured for split tunnel. When we have an outage, no tunnel traffic or internet bound passes.
1) I am connecting from the inside interface when we have an outage, trying from the outside times out. When service is restored, I can connect to the ASDM from either outside or inside.
2) When there is an outage, I can ping both inside and outside interface IP addresses but not the default gateway of my ISP.
The results of a "show conn":
Result of the command: "show conn"
13 in use, 1899 most used
TCP outside 192.168.2.50:3172 inside 192.168.1.50:1025, idle 0:05:38, bytes 72109, flags UIOB
UDP outside 192.42.93.30:53 inside 192.168.1.50:52135, idle 0:00:42, bytes 34, flags -
UDP outside 64.59.160.15:53 inside 192.168.1.50:53444, idle 0:00:48, bytes 34, flags -
UDP outside 64.59.160.13:53 inside 192.168.1.50:53444, idle 0:00:54, bytes 34, flags -
UDP outside 64.59.160.15:53 inside 192.168.1.50:64079, idle 0:00:50, bytes 93, flags -
UDP outside 64.59.160.13:53 inside 192.168.1.50:64079, idle 0:00:50, bytes 93, flags -
TCP outside 192.168.2.50:1025 inside 192.168.1.50:4523, idle 0:05:40, bytes 22109, flags UIO
UDP outside 192.168.0.100:161 inside 192.168.1.69:1032, idle 0:00:42, bytes 1872, flags -
3) attempting to initate traffic fails. the show conn is similar to above...
The attached syslog should show:
the last tunnel transaction
tunnel drops
clear arp
and tunnel/traffic comes back
Also:
A "clear arp" seemes to resolve the issue for a while
I am occasionally running into the 10 user host limit, however the outage seems to occur at any time, even when there is only 1 or 2 hosts on-line.
Thanks again for any assistance
-Jason
06-24-2010 08:04 PM
Hi Jason,
Ok, if a 'clear arp' resolves the issue - even if only for a little while, then that is a good piece of information to know.
Please capture the output of "show arp" (while you can't ping your default gateway), then issue "clear arp", and capture the output of "show arp" again. We are interested to see of the MAC address of your gateway is changing in the ARP cache.
Sincerely,
David
06-30-2010 08:54 PM
I had to re-configure the network so regular NAT traffic was not passed through the ASA so that we didn't loose anymore productivity. I did leave the ASA to handle the VPN tunnel. Since the re-configuration, the ASA has not dropped the tunnel and we have not had any other issues.
At the moment, I don't have anything new to report and I'm reluctant to revert back to the original configuration as we can't handle any more losses in productivity.
So, with that said, I don't have any more useful information to offer.
Thanks again for your suggestions
Jason
07-11-2010 07:33 PM
Hi David,
We began experiencing the same issues as previously mentioned. As per your suggestions, I've done the following:
The pings are to the default gateway:
Result of the command: "ping 24.68.236.1"
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 24.68.236.1, timeout is 2 seconds:
?????
Success rate is 0 percent (0/5)
Result of the command: "show arp"
inside 192.168.1.62 0013.7226.4fca 36
inside 192.168.1.66 0022.1909.a0b8 63
inside 192.168.1.40 0050.ba56.e689 77
inside 192.168.1.103 001c.239c.b1e8 77
inside 192.168.1.100 001e.c956.9444 116
inside 192.168.1.50 0004.5a81.7770 119
outside 24.68.236.1 0030.b8c3.ddc0 0
Result of the command: "clear arp"
The command has been sent to the device
Result of the command: "show arp"
inside 192.168.1.50 0004.5a81.7770 0
inside 192.168.1.40 0050.ba56.e689 10
Result of the command: "ping 24.68.236.1"
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 24.68.236.1, timeout is 2 seconds:
?!!!!
Success rate is 80 percent (4/5), round-trip min/avg/max = 1/10/20 ms
Result of the command: "show arp"
inside 192.168.1.50 0004.5a81.7770 25
inside 192.168.1.40 0050.ba56.e689 35
outside 24.68.236.1 0030.b8c3.ddc0 0
Result of the command: "ping 24.68.236.1"
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 24.68.236.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/8/10 ms
Result of the command: "show arp"
inside 192.168.1.40 0050.ba56.e689 23
inside 192.168.1.66 0022.1909.a0b8 67
inside 192.168.1.50 0004.5a81.7770 168
outside 24.68.236.1 0030.b8c3.ddc0 0
I don't see anything obvious here.
Please let me know if you have any suggestions.
Thanks,
Jason
07-12-2010 08:09 AM
Hi Jason,
OK, this tells us a couple of things:
1) The problem is not caused by the MAC of the Gateway IP changing (as would be the case if there was a duplicate IP for the GW IP)
2) Since the ASA has the correct ARP entry, and is forwarding the packets to the GW, then the problem resides with the GW
3) We know that clearing the ARP table on the ASA resolves the issue, so what does that do?
Well, when we clear the ARP table, and then ping, the ASA will send out an ARP request for the Destination IP, to resolve the MAC. That suceeds, as the ASA is populating it's ARP table again with the GW IP and MAC. Also, this then allows the GW to respond.
Therefore, I would have to speculate that the GW has an incorrect MAC entry for the ASA, and sending the ARP request from the ASA is updating the gateway's entry for the ASAs IP.
One thing that may allow us to detect this is to:
a) Configure a capture on the outside interface, capturing only ARP traffic
b) Set the capture to be a circular-buffer
c) Set the capture size to be like 5 MB
d) When the problem happens, pull the capture off the ASA
Example:
capture out interface outside ethernet-type arp buffer 5000000 circular-buffer
With this, we can look to see if we see another device using your IP at the time the problem occurs.
Sincerely,
David.
07-14-2010 09:11 PM
Hi David,
Thanks for your excellent diagnosis. You were correct in your speculation that the GW had an incorrect MAC for my ASA. I ended up talking with my ISP and they managed to trace the issue to another customer on the same GW who had misconfigured their network equipment and was using our IP address.
Thanks again for your excellent help and persistance with my issue. I'm really grateful for your assistance.
Regards,
Jason
Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: