×

Warning message

  • Cisco Support Forums is in Read Only mode while the site is being migrated.
  • Cisco Support Forums is in Read Only mode while the site is being migrated.

ASA 5505 intermittent drops oubound

Answered Question
Jun 15th, 2010
User Badges:

I'm having a problem with one of two ASA 5505s dropping outbound connections. Effectively the output traffic on the outside interface goes to zero.

The ASAs are configured for a single IPSec tunnel and routing/firewall on either end.

Intermittently one ASA will drop outbound connections. Sometimes it will come back on its own, other times I have to either power cycle the device or log into ASDM and do a system reload. The syslog output doesn't show much other than it dropped the tunnel

It doesn't seem to be system load related since it will do this basically any time. Late at night when the office is closed or during the day.

This config has worked for over 9 months and just recently it started dropping outbound


I'd really appreciate any suggestions.





Result of the command: "show running"

: Saved
:
ASA Version 8.2(1) 
!
hostname ciscoasa
domain-name default.domain.invalid
enable password Sl7iFbMSQUs45M7. encrypted
passwd Sl7iFbMSQUs45M7. encrypted
names
dns-guard
!
interface Vlan1
nameif inside
security-level 100
ip address 192.168.1.1 255.255.255.0 
!
interface Vlan2
nameif outside
security-level 0
ip address 24.68.xxx.xxx 255.255.252.0 
!
interface Ethernet0/0
switchport access vlan 2
!
interface Ethernet0/1
!
interface Ethernet0/2
!
interface Ethernet0/3
!
interface Ethernet0/4
!
interface Ethernet0/5
!
interface Ethernet0/6
!
interface Ethernet0/7
!
boot system disk0:/asa821-k8.bin
boot system disk0:/asa724-k8.bin
ftp mode passive
clock timezone PST -8
clock summer-time PDT recurring
dns domain-lookup inside
dns domain-lookup outside
dns server-group DefaultDNS
name-server 64.59.160.13
name-server 64.59.160.15
domain-name default.domain.invalid
access-list inside_nat0_outbound extended permit ip 192.168.1.0 255.255.255.0 192.168.2.0 255.255.255.0 
access-list outside_access_in extended permit tcp any interface outside eq pptp 
access-list outside_cryptomap extended permit ip 192.168.1.0 255.255.255.0 192.168.2.0 255.255.255.0 
access-list Internal_ICMP extended permit icmp 192.168.2.0 255.255.255.0 192.168.1.0 255.255.255.0 
pager lines 24
logging enable
logging timestamp
logging monitor warnings
logging trap debugging
logging asdm debugging
logging recipient-address
logging host inside 192.168.1.50
logging debug-trace
mtu inside 1500
mtu outside 1500
ip verify reverse-path interface outside
icmp unreachable rate-limit 1 burst-size 1
asdm image disk0:/asdm-621.bin
asdm history enable
arp timeout 14400
global (outside) 1 interface
nat (inside) 0 access-list inside_nat0_outbound
nat (inside) 1 0.0.0.0 0.0.0.0
static (inside,outside) tcp interface pptp 192.168.1.50 pptp netmask 255.255.255.255 
access-group outside_access_in in interface outside
route outside 0.0.0.0 0.0.0.0 24.68.238.1 1
timeout xlate 3:00:00
timeout conn 1:00:00 half-closed 0:10:00 udp 0:02:00 icmp 0:00:02
timeout sunrpc 0:10:00 h323 0:05:00 h225 1:00:00 mgcp 0:05:00 mgcp-pat 0:05:00
timeout sip 0:30:00 sip_media 0:02:00 sip-invite 0:03:00 sip-disconnect 0:02:00
timeout sip-provisional-media 0:02:00 uauth 0:05:00 absolute
timeout tcp-proxy-reassembly 0:01:00
dynamic-access-policy-record DfltAccessPolicy
http server enable
http 192.168.1.0 255.255.255.0 inside
http 96.50.xxx.xxx 255.255.255.255 outside
no snmp-server location
no snmp-server contact
snmp-server enable traps snmp authentication linkup linkdown coldstart
crypto ipsec transform-set ESP-AES-256-MD5 esp-aes-256 esp-md5-hmac 
crypto ipsec transform-set ESP-DES-SHA esp-des esp-sha-hmac 
crypto ipsec transform-set ESP-DES-MD5 esp-des esp-md5-hmac 
crypto ipsec transform-set ESP-AES-192-MD5 esp-aes-192 esp-md5-hmac 
crypto ipsec transform-set ESP-3DES-MD5 esp-3des esp-md5-hmac 
crypto ipsec transform-set ESP-AES-256-SHA esp-aes-256 esp-sha-hmac 
crypto ipsec transform-set ESP-AES-128-SHA esp-aes esp-sha-hmac 
crypto ipsec transform-set ESP-AES-192-SHA esp-aes-192 esp-sha-hmac 
crypto ipsec transform-set ESP-AES-128-MD5 esp-aes esp-md5-hmac 
crypto ipsec transform-set ESP-3DES-SHA esp-3des esp-sha-hmac 
crypto ipsec security-association lifetime seconds 28800
crypto ipsec security-association lifetime kilobytes 4608000
crypto map outside_map1 2 match address outside_cryptomap
crypto map outside_map1 2 set pfs group1
crypto map outside_map1 2 set peer 96.50.xxx.xxx 
crypto map outside_map1 2 set transform-set ESP-AES-128-SHA ESP-AES-128-MD5 ESP-AES-192-SHA ESP-AES-192-MD5 ESP-AES-256-SHA ESP-AES-256-MD5 ESP-3DES-SHA ESP-3DES-MD5 ESP-DES-SHA ESP-DES-MD5
crypto map outside_map1 2 set reverse-route
crypto map outside_map1 interface outside
crypto isakmp enable inside
crypto isakmp enable outside
crypto isakmp policy 10
authentication pre-share
encryption 3des
hash sha
group 2
lifetime 86400
telnet timeout 5
ssh timeout 5
console timeout 0
dhcpd dns 64.59.160.13 64.59.160.15
dhcpd auto_config outside
!
dhcpd address 192.168.1.2-192.168.1.33 inside
!

threat-detection basic-threat
threat-detection scanning-threat shun
threat-detection statistics
threat-detection statistics tcp-intercept rate-interval 30 burst-rate 400 average-rate 200
webvpn
group-policy DfltGrpPolicy attributes
vpn-tunnel-protocol IPSec l2tp-ipsec 
tunnel-group 96.50.xxx.xxx type ipsec-l2l
tunnel-group 96.50.xxx.xxx ipsec-attributes
pre-shared-key *
!
class-map inspection_default
match default-inspection-traffic
!
!
policy-map type inspect dns preset_dns_map
parameters
  message-length maximum 512
policy-map global_policy
class inspection_default
  inspect dns preset_dns_map 
  inspect ftp 
  inspect h323 h225 
  inspect h323 ras 
  inspect rsh 
  inspect rtsp 
  inspect esmtp 
  inspect sqlnet 
  inspect skinny  
  inspect sunrpc 
  inspect xdmcp 
  inspect sip  
  inspect netbios 
  inspect tftp 
  inspect pptp 
!
service-policy global_policy global
prompt hostname context 
Cryptochecksum:6d3be823caf96c4e5d264e440d64c609
: end

Correct Answer by David White about 7 years 1 month ago

Hi Jason,


OK, this tells us a couple of things:

1) The problem is not caused by the MAC of the Gateway IP changing (as would be the case if there was a duplicate IP for the GW IP)

2) Since the ASA has the correct ARP entry, and is forwarding the packets to the GW, then the problem resides with the GW

3) We know that clearing the ARP table on the ASA resolves the issue, so what does that do?

Well, when we clear the ARP table, and then ping, the ASA will send out an ARP request for the Destination IP, to resolve the MAC.  That suceeds, as the ASA is populating it's ARP table again with the GW IP and MAC.  Also, this then allows the GW to respond.


Therefore, I would have to speculate that the GW has an incorrect MAC entry for the ASA, and sending the ARP request from the ASA is updating the gateway's entry for the ASAs IP.


One thing that may allow us to detect this is to:

a) Configure a capture on the outside interface, capturing only ARP traffic

b) Set the capture to be a circular-buffer

c) Set the capture size to be like 5 MB

d) When the problem happens, pull the capture off the ASA


Example:

  capture out interface outside ethernet-type arp buffer 5000000 circular-buffer


With this, we can look to see if we see another device using your IP at the time the problem occurs.


Sincerely,


David.

  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 5 (1 ratings)
Loading.
David White Fri, 06/18/2010 - 11:42
User Badges:
  • Cisco Employee,

Hi Jason,


When you have the outbound drops - it sounds like no traffic egresses the outside interface? Or is it only tunneled traffic?


1) When you connect via ASDM, are you connecting to the outside interface, or the inside?


2) Access the ASA when you have the problem, and verify you can ping the adjacent devices out both interfaces.  If that works, then try initiating traffic through the ASA - and record the "show conn" output as well as the syslogs. 


3) If the tunnel is down, attempt to send traffic across the tunnel, and see if the tunnel comes up.  Again, capture the 'show conn' and syslog outputs.


Thanks,


David.

jasoncobham Tue, 06/22/2010 - 18:10
User Badges:

Hi David,


Thanks for your reply.


The router is configured for split tunnel. When we have an outage, no tunnel traffic or internet bound passes.


1) I am connecting from the inside interface when we have an outage, trying from the outside times out. When service is restored, I can connect to the ASDM from either outside or inside.


2) When there is an outage, I can ping both inside and outside interface IP addresses but not the default gateway of my ISP.

The results of a "show conn":


Result of the command: "show conn"


13 in use, 1899 most used
TCP outside 192.168.2.50:3172 inside 192.168.1.50:1025, idle 0:05:38, bytes 72109, flags UIOB
UDP outside 192.42.93.30:53 inside 192.168.1.50:52135, idle 0:00:42, bytes 34, flags -
UDP outside 64.59.160.15:53 inside 192.168.1.50:53444, idle 0:00:48, bytes 34, flags -
UDP outside 64.59.160.13:53 inside 192.168.1.50:53444, idle 0:00:54, bytes 34, flags -
UDP outside 64.59.160.15:53 inside 192.168.1.50:64079, idle 0:00:50, bytes 93, flags -
UDP outside 64.59.160.13:53 inside 192.168.1.50:64079, idle 0:00:50, bytes 93, flags -
TCP outside 192.168.2.50:1025 inside 192.168.1.50:4523, idle 0:05:40, bytes 22109, flags UIO
UDP outside 192.168.0.100:161 inside 192.168.1.69:1032, idle 0:00:42, bytes 1872, flags -


3) attempting to initate traffic fails. the show conn is similar to above...


The attached syslog should show:

the last tunnel transaction

tunnel drops

clear arp

and tunnel/traffic comes back




Also:

A "clear arp" seemes to resolve the issue for a while

I am occasionally running into the 10 user host limit, however the outage seems to occur at any time, even when there is only 1 or 2 hosts on-line.



Thanks again for any assistance


-Jason

David White Thu, 06/24/2010 - 20:04
User Badges:
  • Cisco Employee,

Hi Jason,


Ok, if a 'clear arp' resolves the issue - even if only for a little while, then that is a good piece of information to know.


Please capture the output of "show arp" (while you can't ping your default gateway), then issue "clear arp", and capture the output of "show arp" again.  We are interested to see of the MAC address of your gateway is changing in the ARP cache.


Sincerely,


David

jasoncobham Wed, 06/30/2010 - 20:54
User Badges:

I had to re-configure the network so regular NAT traffic was not passed through the ASA so that we didn't loose anymore productivity. I did leave the ASA to handle the VPN tunnel. Since the re-configuration, the ASA has not dropped the tunnel and we have not had any other issues.


At the moment, I don't have anything new to report and I'm reluctant to revert back to the original configuration as we can't handle any more losses in productivity.


So, with that said, I don't have any more useful information to offer.


Thanks again for your suggestions


Jason

jasoncobham Sun, 07/11/2010 - 19:33
User Badges:

Hi David,


We began experiencing the same issues as previously mentioned. As per your suggestions, I've done the following:


The pings are to the default gateway:


Result of the command: "ping 24.68.236.1"


Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 24.68.236.1, timeout is 2 seconds:
?????
Success rate is 0 percent (0/5)




Result of the command: "show arp"


    inside 192.168.1.62 0013.7226.4fca 36
    inside 192.168.1.66 0022.1909.a0b8 63
    inside 192.168.1.40 0050.ba56.e689 77
    inside 192.168.1.103 001c.239c.b1e8 77
    inside 192.168.1.100 001e.c956.9444 116
    inside 192.168.1.50 0004.5a81.7770 119
    outside 24.68.236.1 0030.b8c3.ddc0 0




Result of the command: "clear arp"


The command has been sent to the device




Result of the command: "show arp"


    inside 192.168.1.50 0004.5a81.7770 0
    inside 192.168.1.40 0050.ba56.e689 10




Result of the command: "ping 24.68.236.1"


Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 24.68.236.1, timeout is 2 seconds:
?!!!!
Success rate is 80 percent (4/5), round-trip min/avg/max = 1/10/20 ms




Result of the command: "show arp"


    inside 192.168.1.50 0004.5a81.7770 25
    inside 192.168.1.40 0050.ba56.e689 35
    outside 24.68.236.1 0030.b8c3.ddc0 0




Result of the command: "ping 24.68.236.1"


Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 24.68.236.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/8/10 ms




Result of the command: "show arp"


    inside 192.168.1.40 0050.ba56.e689 23
    inside 192.168.1.66 0022.1909.a0b8 67
    inside 192.168.1.50 0004.5a81.7770 168
    outside 24.68.236.1 0030.b8c3.ddc0 0




I don't see anything obvious here.

Please let me know if you have any suggestions.


Thanks,


Jason

Correct Answer
David White Mon, 07/12/2010 - 08:09
User Badges:
  • Cisco Employee,

Hi Jason,


OK, this tells us a couple of things:

1) The problem is not caused by the MAC of the Gateway IP changing (as would be the case if there was a duplicate IP for the GW IP)

2) Since the ASA has the correct ARP entry, and is forwarding the packets to the GW, then the problem resides with the GW

3) We know that clearing the ARP table on the ASA resolves the issue, so what does that do?

Well, when we clear the ARP table, and then ping, the ASA will send out an ARP request for the Destination IP, to resolve the MAC.  That suceeds, as the ASA is populating it's ARP table again with the GW IP and MAC.  Also, this then allows the GW to respond.


Therefore, I would have to speculate that the GW has an incorrect MAC entry for the ASA, and sending the ARP request from the ASA is updating the gateway's entry for the ASAs IP.


One thing that may allow us to detect this is to:

a) Configure a capture on the outside interface, capturing only ARP traffic

b) Set the capture to be a circular-buffer

c) Set the capture size to be like 5 MB

d) When the problem happens, pull the capture off the ASA


Example:

  capture out interface outside ethernet-type arp buffer 5000000 circular-buffer


With this, we can look to see if we see another device using your IP at the time the problem occurs.


Sincerely,


David.

jasoncobham Wed, 07/14/2010 - 21:11
User Badges:

Hi David,


Thanks for your excellent diagnosis. You were correct in your speculation that the GW had an incorrect MAC for my ASA. I ended up talking with my ISP and they managed to trace the issue to another customer on the same GW who had misconfigured their network equipment and was using our IP address.


Thanks again for your excellent help and persistance with my issue. I'm really grateful for your assistance.


Regards,


Jason

Actions

This Discussion