ASA 5510 Extremely slow Internet speeds

[mitchell helton]

We have an ASA 5510 and are experiencing unbelievably slow speeds.  I noticed a problem last Thursday with users complaining of slow speeds and realized our interface had a ton of errors and was running at half duplex.  I contacted the ISP (we are connected to their 3750) and they swore up and down they were set to full.  So they had me switch to full and the interface shut down.  I asked them to switch to auto and the interface came back up and we went to full, and of course the errors and colisions stopped.  However the errors and packet drops have not stopped.  The ISP sent out a technician and they determined it wasn't a problem on their end by plugging in a laptop and testing the speed--that worked fine.  Eventually I plugged in a Sonicwall and bypassed the ASA completely and that worked fine.  We plugged the ASA back in and we we went back to dropping packets.  I put an old config on the ASA and oddly enough it seemed to have fixed the problem but we were still dropping packets.  So I put the most recent config back on and that worked fine up until today.  We're back in the some boat we were last week.  So my first question is when I do a show int and see packets dropped - is that normal because of ACLs etc, or would that be show in another place?  Is it possible it's a config issue with the ISPs router?  Here's an output of show int and show asp drop:

HQ-ASA# show asp drop

Frame drop:
  Flow is denied by configured rule (acl-drop)                              3366
  NAT-T keepalive message (natt-keepalive)                                   423
  First TCP packet not SYN (tcp-not-syn)                                     406
  TCP failed 3 way handshake (tcp-3whs-failed)                               135
  TCP RST/FIN out of order (tcp-rstfin-ooo)                                  462
  TCP SYNACK on established conn (tcp-synack-ooo)                             46
  TCP packet SEQ past window (tcp-seq-past-win)                               50
  TCP invalid ACK (tcp-invalid-ack)                                            9
  TCP Out-of-Order packet buffer full (tcp-buffer-full)                       29
  TCP Out-of-Order packet buffer timeout (tcp-buffer-timeout)               6634
  TCP RST/SYN in window (tcp-rst-syn-in-win)                                   9
  TCP dup of packet in Out-of-Order queue (tcp-dup-in-queue)                2223
  TCP packet failed PAWS test (tcp-paws-fail)                                 43
  DNS Inspect id not matched (inspect-dns-id-not-matched)                     31
  Dropped pending packets in a closed socket (np-socket-closed)               49

Last clearing: 10:38:04 EST Jan 17 2012 by admin

Flow drop:
  NAT failed (nat-failed)                                                     56
  Inspection failure (inspect-fail)                                            4

Last clearing: 10:38:04 EST Jan 17 2012 by admin
HQ-ASA# show int e0/0
Interface Ethernet0/0 "WAN", is up, line protocol is up
  Hardware is i82546GB rev03, BW 1000 Mbps, DLY 10 usec
        Full-Duplex(Full-duplex), 10 Mbps(10 Mbps)
        Input flow control is unsupported, output flow control is unsupported
        Description: WAN connection to Internet
        MAC address , MTU 1500
        IP address , subnet mask 255.255.255.0
        1273672 packets input, 1436097018 bytes, 0 no buffer
        Received 483 broadcasts, 0 runts, 0 giants
        481652 input errors, 481652 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
        0 L2 decode drops
        977850 packets output, 262154054 bytes, 0 underruns
        0 pause output, 0 resume output
        0 output errors, 0 collisions, 0 interface resets
        0 late collisions, 0 deferred
        0 input reset drops, 0 output reset drops, 0 tx hangs
        input queue (blocks free curr/low): hardware (255/236)
        output queue (blocks free curr/low): hardware (255/0)
  Traffic Statistics for "WAN":
        1273649 packets input, 1412506101 bytes
        977850 packets output, 241223057 bytes
        13021 packets dropped
      1 minute input rate 259 pkts/sec,  245821 bytes/sec
      1 minute output rate 222 pkts/sec,  38371 bytes/sec
      1 minute drop rate, 2 pkts/sec
      5 minute input rate 359 pkts/sec,  419612 bytes/sec
      5 minute output rate 257 pkts/sec,  31775 bytes/sec
      5 minute drop rate, 2 pkts/sec

I have not made any configuration changes to the ASA ina  couple of months.  The interface counters were cleared about 45 minutes ago if that's any idea how quickly the errors/packet drops are adding up.  Please help--thanks!

[danielnunes]

Hi,

first of all, be sure that your cable are OK, try to change your cable to new cable.

Packet drop could be many things like multicast packets or broadcast packets hiting the ASA.

[mitchell helton]

Thanks for the reply... I've bypassed the punchdown and used 2 different cables and reproduced the same results.

[Julio Carvajal · Accepted solution]

Hello Mitch,

A speed or duplex mismatch is most frequently  revealed when error counters on the interfaces in question increase. The  most common errors are frame, cyclic redundancy checks (CRCs), and  runts. If these values increment on your interface, either a  speed/duplex mismatch or a cabling issue occurs. You must resolve this  issue before you continue.

Please rate helpful posts.

Julio

[Rishabh Seth]

Hi Mitchell,

From the ASP drop counter I see that the major contributor in asp drops apart from acl-drop(3326)

is out-of order and duplicate packets reaching ASA.

TCP Out-of-Order packet buffer timeout (tcp-buffer-timeout)       6634

TCP dup of packet in Out-of-Order queue (tcp-dup-in-queue)    2223

>> Now according to me you should investigate with the ISP and check why there are so many out-of-order and duplicate packets in the network. 

>> While testing the speeds with laptop you may not see much problem because the buffer that you  have on a laptop will be more that what you have on firewall. 

>> It would be better if you capture traffic on the laptop directly plugged and check if there are any out-of-order and duplicate packets.

>>Try to get this resolved on the ISP end.

Hope it helps!!

[joshking1]

Hi Mitchell,

Did you eventually get a solution for this or find the root cause of the the problem?

Thanks.

Josh

[Rishabh Seth]

Hi Mitchell,

From the ASP drop counter I see that the major contributor in asp drops apart from acl-drop(3326)

is out-of order and duplicate packets reaching ASA.

TCP Out-of-Order packet buffer timeout (tcp-buffer-timeout)       6634

TCP dup of packet in Out-of-Order queue (tcp-dup-in-queue)    2223

>> Now according to me you should investigate with the ISP and check why there are so many out-of-order and duplicate packets in the network. 

>> While testing the speeds with laptop you may not see much problem because the buffer that you  have on a laptop will be more that what you have on firewall. 

>> It would be better if you capture traffic on the laptop directly plugged and check if there are any out-of-order and duplicate packets.

>>Try to get this resolved on the ISP end.

Hope it helps!!