Solved: ASA 5505 Random Connection Drops

Michichael · ‎07-30-2010

Ok folks, got a puzzle for ya here. First, the background info:

Our network consists of Catalyst 2950 Managed switches connected to an MPLS router as our primary point of service. We also had a PIX506E connected to an AT&T DSL modem, which acted as a local VPN endpoint, as well as a site to site tunnel back to corporate - the route tracking for failover performed at the switch level.

We never had any problems with it. We recently replaced the PIX with an ASA - same infrastructure. The tunnel comes up, failover works better than before (previously we'd have to reboot the PIX to drop the tunnel for failback). However, now we're having a very strange issue of the ASA5505 dropping connections randomly.

Example: I'm remote desktoped to my home server. At random, the connection will say that it has been lost, then reconnect. On the ASA I see the connection torn down with TCP RESET-I, and two seconds later, rebuilt.

Example2: From another workstation, I attempt to watch a youtube stream. The video loads partially, then stops. Refreshing the video causes it to reload, and it sometimes loads or stops at a different spot - earlier or later than the previous stop. It's random.

I've seen no established root cause - connections have lasted between 30 seconds and 30 minutes before randomly dropping.

Troubleshooting so far: I've dropped the MTU to 1492 to account for PPPoE encapsulation, however that has not fixed the problem. The ACL's and VPN setup are identical to the previous PIX. The AT&T Router has the ASA set to a public IP address (No NAT) and has the firewall disabled for that traffic.

This set up worked fine under the PIX. The ASA is running 8.2(2).

I've cleared my asp drop to see if there's a specific packet drop category occuring. Immediately after clearing, l2_acl and acl-drop both start incrementing, but the connection stream is fine. When a connection stream drops, nothing new is added to the list. Occassionaly TCP Failed 3 way handshake and slowpath secruity check failed packets show up, but they don't seem related to the drops.

Another side note - normal connection teardowns appear to be TCP RESET-I as well, from just basic web browsing. At least they appear to be judging from my coworker's web browsing habits...

Anyway, like I said - seems random, no root cause established, and honestly I'm baffled. If anyone's got an idea, you let me know...

Here's my config, vpn, passwords, etc removed:

: Saved
:
ASA Version 8.2(2)
!
names
name 192.168.240.0 corporate-network
!
interface Vlan2
mac-address 4c75.6115.0d01
nameif outside
security-level 0
ip address *.*.*.* 255.255.255.248
!
interface Vlan20
nameif inside
security-level 100
ip address 10.20.12.10 255.255.255.0
!
interface Vlan91
mac-address 4c75.6115.0d91
no forward interface Vlan20
nameif dmz
security-level 50
ip address 10.91.0.252 255.255.255.0
!
interface Ethernet0/0
switchport access vlan 2
!
interface Ethernet0/1
switchport access vlan 91
!
interface Ethernet0/2
switchport access vlan 20
!
interface Ethernet0/3
!
interface Ethernet0/4
!
interface Ethernet0/5
!
interface Ethernet0/6
!
interface Ethernet0/7
!
boot system disk0:/asa822-k8.bin
ftp mode passive
clock timezone PST -8
clock summer-time PDT recurring
same-security-traffic permit intra-interface
object-group network DM_INLINE_NETWORK_1
network-object 10.20.11.0 255.255.255.0
network-object corporate-network 255.255.255.0
object-group network DM_INLINE_NETWORK_2
network-object 10.20.11.0 255.255.255.0
network-object corporate-network 255.255.255.0
access-list outside_access_in extended permit icmp any any
access-list outside_1_cryptomap extended permit ip 10.20.12.0 255.255.255.0 object-group DM_INLINE_NETWORK_2
access-list inside_nat0_outbound extended permit ip 10.20.12.0 255.255.255.0 object-group DM_INLINE_NETWORK_2
access-list inside_nat0_outbound extended permit ip 10.20.12.0 255.255.255.0 10.90.10.0 255.255.255.192
access-list inside_nat0_outbound extended permit ip 10.20.11.0 255.255.255.0 10.90.10.0 255.255.255.192
access-list inside_nat0_outbound extended permit ip corporate-network 255.255.255.0 10.90.10.0 255.255.255.192
access-list TASV_splitTunnelAcl standard permit 10.20.12.0 255.255.255.0
access-list TASV_splitTunnelAcl standard permit 10.20.11.0 255.255.255.0
access-list TASV_splitTunnelAcl standard permit corporate-network 255.255.255.0
access-list DefaultRAGroup_splitTunnelAcl standard permit 10.20.12.0 255.255.255.0
pager lines 24
logging enable
logging trap warnings
logging asdm debugging
logging host inside 10.20.12.17
logging permit-hostdown
no logging message 405001
no logging message 710005
logging message 106026 level warnings
mtu outside 1492
mtu inside 1500
mtu dmz 1500
ip local pool VpnPool 10.90.10.25-10.90.10.60 mask 255.255.255.0
icmp unreachable rate-limit 1 burst-size 1
asdm image disk0:/asdm-625-53.bin
no asdm history enable
arp timeout 14400
global (outside) 101 interface
nat (inside) 0 access-list inside_nat0_outbound
nat (inside) 101 10.20.12.0 255.255.255.0
access-group outside_access_in in interface outside
route outside 0.0.0.0 0.0.0.0 *.*.*.* 1
timeout xlate 3:00:00
timeout conn 1:00:00 half-closed 0:10:00 udp 0:02:00 icmp 0:00:02
timeout sunrpc 0:10:00 h323 0:05:00 h225 1:00:00 mgcp 0:05:00 mgcp-pat 0:05:00
timeout sip 0:30:00 sip_media 0:02:00 sip-invite 0:03:00 sip-disconnect 0:02:00
timeout sip-provisional-media 0:02:00 uauth 0:05:00 absolute
timeout tcp-proxy-reassembly 0:01:00
dynamic-access-policy-record DfltAccessPolicy
aaa authentication ssh console LOCAL
http server enable
http 10.20.12.214 255.255.255.255 inside
http 10.20.12.182 255.255.255.255 inside
no snmp-server location
no snmp-server contact
snmp-server enable traps snmp authentication linkup linkdown coldstart
fragment size 300 inside
sysopt connection tcpmss 0
no service resetoutbound interface outside
no service resetoutbound interface inside
crypto ipsec transform-set ESP-AES-256-MD5 esp-aes-256 esp-md5-hmac
crypto ipsec transform-set ESP-DES-SHA esp-des esp-sha-hmac
crypto ipsec transform-set ESP-DES-MD5 esp-des esp-md5-hmac
crypto ipsec transform-set ESP-AES-192-MD5 esp-aes-192 esp-md5-hmac
crypto ipsec transform-set ESP-AES-256-SHA esp-aes-256 esp-sha-hmac
crypto ipsec transform-set ESP-AES-128-SHA esp-aes esp-sha-hmac
crypto ipsec transform-set ESP-AES-192-SHA esp-aes-192 esp-sha-hmac
crypto ipsec transform-set ESP-AES-128-MD5 esp-aes esp-md5-hmac
crypto ipsec transform-set ESP-3DES-MD5 esp-3des esp-md5-hmac
crypto ipsec transform-set ESP-3DES-SHA esp-3des esp-sha-hmac
crypto ipsec transform-set TRANS_ESP_3DES_SHA esp-3des esp-sha-hmac
crypto ipsec transform-set TRANS_ESP_3DES_SHA mode transport
crypto ipsec security-association lifetime seconds 28800
crypto ipsec security-association lifetime kilobytes 4608000
crypto dynamic-map SYSTEM_DEFAULT_CRYPTO_MAP 10 set transform-set TRANS_ESP_3DES_SHA ESP-AES-128-SHA ESP-AES-128-MD5 ESP-AES-192-SHA ESP-AES-192-MD5 ESP-AES-256-SHA ESP-AES-256-MD5 ESP-3DES-SHA ESP-3DES-MD5 ESP-DES-SHA ESP-DES-MD5
crypto dynamic-map SYSTEM_DEFAULT_CRYPTO_MAP 65535 set transform-set ESP-AES-128-SHA ESP-AES-128-MD5 ESP-AES-192-SHA ESP-AES-192-MD5 ESP-AES-256-SHA ESP-AES-256-MD5 ESP-3DES-SHA ESP-3DES-MD5 ESP-DES-SHA ESP-DES-MD5
crypto map outside_map2 65535 ipsec-isakmp dynamic SYSTEM_DEFAULT_CRYPTO_MAP
crypto map outside_map2 interface outside
crypto isakmp enable outside
crypto isakmp policy 5
authentication pre-share
encryption 3des
hash md5
group 2
lifetime 86400
crypto isakmp policy 10
authentication pre-share
encryption 3des
hash sha
group 2
lifetime 86400
telnet timeout 5
ssh 10.20.12.25 255.255.255.255 inside
ssh 10.20.12.214 255.255.255.255 inside
ssh timeout 20
console timeout 0
dhcp-client update dns server none

threat-detection basic-threat
no threat-detection statistics access-list
no threat-detection statistics tcp-intercept
ntp server 10.20.12.14 source inside prefer
webvpn
anyconnect-essentials
!
!
!
policy-map type inspect dns default_DNS
parameters
message-length maximum 768
no nat-rewrite
id-randomization
id-mismatch action log
policy-map type inspect dcerpc default_DCERPC
parameters
timeout pinhole 0:01:00
!
Cryptochecksum:18c6fab94f08788ae39661317f6888c2

Magnus Mortensen · ‎08-03-2010

It may make sense to also change the MSS on the firewall in order to keep TCP packets small enough to avoid fragmentation. If you have dropped the MTU down to 1492 an MSS of the default of 1380 (which i see has been changed in the configuration for some reason) should help keep TCP in check as well. You can set this value with the command 'sysopt connection tcpmss 1380'. 1380 would leave 112 bytes available for headers etc (1492-1380).

- Magnus

View solution in original post

marbrow2 · ‎07-30-2010

I'm not sure why some device is sending messages with the reset flag. Can you list the log messages (w/o the public ip's)?

I would suggest that you use a combination of captures and logs to get to the bottom of the issue.

-- Get the issue to occur

-- Set your logging to "logging trap debugging" and then grep your logs for the internal ip address that you are having the issue (or use ASDM live logging)

-- Set up captures on the inside

-- See what traffic is sent just prior to the reset flags being sent, and see what device is sourcing the reset.

access-list capin permit ip any host ip_address_of_host

access-list capin permit ip any 10.20.12.10

capture capin access-list capin interface inside

show capture capin

Michichael · ‎08-03-2010

Mark,

Dealing with other issues at the moment, but I will post the results as soon as I have time to test it. Thank you.

JORGE RODRIGUEZ · ‎08-03-2010

This sounds like physical layer problem but could be wrong, beside what Mark has asked you to do , in addition, could you look at your interfaces to rule out you are not dropping packets due to transmission mismatch .. same applies for your local hosts switchport and nic settings, always good to rule out the physical and move up the latter.

Regards

Jorge Rodriguez

Michichael · ‎08-03-2010

Well, so far so good - I haven't had any random drops since correcting the MTU (DSL requires an MTU of 1492, since 8 bytes are used for PPPoE Encapsulation). I think that might have been our culprit, but I will leave this open for a few more days as I test.

For anyone having this problem - set your outside interface to an MTU of 1492 if you're on DSL, especially on 2Wire modems/routers.

Magnus Mortensen · ‎08-03-2010

It may make sense to also change the MSS on the firewall in order to keep TCP packets small enough to avoid fragmentation. If you have dropped the MTU down to 1492 an MSS of the default of 1380 (which i see has been changed in the configuration for some reason) should help keep TCP in check as well. You can set this value with the command 'sysopt connection tcpmss 1380'. 1380 would leave 112 bytes available for headers etc (1492-1380).

- Magnus

Michichael · ‎08-04-2010

Thank you for the suggestion. I have not had to implement it - it appears the MTU issue fixed everything - I have had no further drops on the connection.

Once again, the solution was to set my MTU to 1492 due to being on a DSL Connection.

Thank you everyone.

yahyajaber · ‎07-31-2010

Same Problem but it happens when i download too!

bbiandov · ‎12-01-2012

Good call on the 2Wire routers, pesky PPPoE. One must also clarify that the PPPoE session lives at the Cisco ASA not at the DSL gear, which is some times what they want you to do; may be if the session lives at the DSL box they internally know about this MTU issue and control it.

Long story short, the MTU did work here as well:

mtu outside 1492

mtu inside 1500