ACE Dup ACK and TCP Out-of-order

ian.richards · ‎06-21-2010

Hi,

I have a pair of FT ACE 4710 offloading https traffic to a couple of webservers. We are seeing very high network utilisation when I capture the client facing port of the active ACE. There appears to alot of duplicate ACKs and TCP out-of-order packets (as shown by wireshark). Does anyone know if this is a problem with the ACE or "normal"

Thanks

david.stout · ‎07-06-2010

I've seen some similar behaviour with the ACE Module and Apache webservers. To mitigate this I've configured the following which seems to work.

On the ACE Module

parameter-map type http ALL-HEADERS
persistence-rebalance

parameter-map type connection TCP-OPTIONS
set tcp syn-retry 5
tcp-options timestamp allow

policy-map multi-match test-policy
class http-vip
    loadbalance vip inservice
    loadbalance policy http-test-pm
    loadbalance vip icmp-reply active
    appl-parameter http advanced-options ALL-HEADERS
    connection advanced-options TCP-OPTIONS

On Apache here are the two test results with keepalive on and off

########

httpd.conf

########

KeepAlive Off
MaxKeepAliveRequests 1024
KeepAliveTimeout 30

########

MK-ACE01/001# show serverfarm MK-FARM-sf
serverfarm     : MK-FARM-sf, type: HOST
total rservers : 8
---------------------------------
                                                ----------connections-----------
       real                  weight state        current    total      failures
   ---+---------------------+------+------------+----------+----------+---------
   rserver: MK-HOST10
       10.10.1.10:0          8      OPERATIONAL 321        510863     16442
   rserver: MK-HOST11
       10.10.1.11:0          8      OPERATIONAL 304        512718     16276
   rserver: MK-HOST12
       10.10.1.12:0          8      OPERATIONAL 286        524207     17257
   rserver: MK-HOST13
       10.10.1.13:0          8      OPERATIONAL 291        516987     16626
   rserver: MK-HOST14
       10.10.1.14:0          8      OPERATIONAL 291        513016     16594
   rserver: MK-HOST15
       10.10.1.15:0          8      OPERATIONAL 311        510177     16434
   rserver: MK-HOST16
       10.10.1.16:0          8      OPERATIONAL 345        516340     16708
   rserver: MK-HOST17
       10.10.1.17:0          8      OPERATIONAL 282        513046     16418

#########

httpd.conf

#########
KeepAlive On
MaxKeepAliveRequests 1024
KeepAliveTimeout 30

#########

MK-ACE01/001# show serverfarm MK-FARM-sf
serverfarm     : MK-FARM-sf, type: HOST
total rservers : 8
---------------------------------
                                                ----------connections-----------
       real                  weight state        current    total      failures
   ---+---------------------+------+------------+----------+----------+---------
   rserver: MK-HOST10
       10.10.1.10:0          8      OPERATIONAL 0          553        0
   rserver: MK-HOST11
       10.10.1.11:0          8      OPERATIONAL 0          551        0
   rserver: MK-HOST12
       10.10.1.12:0          8      OPERATIONAL 0          552        0
   rserver: MK-HOST13
       10.10.1.13:0          8      OPERATIONAL 0          555        0
   rserver: MK-HOST14
       10.10.1.14:0          8      OPERATIONAL 0          554        0
   rserver: MK-HOST15
       10.10.1.15:0          8      OPERATIONAL 0          551        0
   rserver: MK-HOST16
       10.10.1.16:0          8      OPERATIONAL 0          550        0
   rserver: MK-HOST17
       10.10.1.17:0          8      OPERATIONAL 0          550        0

This seems to of reduced the large number or re-transmits and dup-acks.

ian.richards · ‎07-08-2010

Hi,

Thanks for this. Unfortunately I have missed the boat to test this until August now But I will add to the list.

I was wondering if normalisation could be complicit as well, have you had experience of this?

Regards

Ian Richards

**********************************************************************

COMPUTACENTER PLC is registered in England and Wales with the registered number 03110569. Its registered office is at Hatfield Business Park, Hatfield Avenue, Hatfield, Hertfordshire AL10 9TW

COMPUTACENTER (UK) Limited is registered in England and Wales with the registered number 01584718. Its registered office is at Hatfield Business Park, Hatfield Avenue, Hatfield, Hertfordshire AL10 9TW

COMPUTACENTER (Mid-Market) Limited is registered in England and Wales with the registered number 3434654. Its registered office is at Hatfield Business Park, Hatfield Avenue, Hatfield, Hertfordshire AL10 9TW

COMPUTACENTER (FMS) Limited is registered in England and Wales with the registered number 3798091. Its registered office is at Hatfield Business Park, Hatfield Avenue, Hatfield, Hertfordshire AL10 9TW

The contents of this email are intended for the named addressee only.

It contains information which may be confidential and which may also be privileged.

Unless you are the named addressee (or authorised to receive mail for the addressee) you may not copy or use it, or disclose it to anyone else.

If you receive it in error please notify us immediately and then destroy it.

Computacenter information is available from: http://www.computacenter.com

david.stout · ‎07-08-2010

During my recent testing I found that normalisation did not affect the results. If you turn off normalisation and things improve it's a false economy because something will still be fundamentally broken under the hood.

Although as a rule of thumb .....

If there is a firewall in front of the ACE then it's ok to turn off normalisation. If the ACE is acting as a firewall / loadbalancer then it's not ok to turn off normalisation.

Hope that helps,

Dave Stout

ian.richards · ‎07-08-2010

Hi,

Thanks for the input. We do have a firewall in front of the ACE, so might be worth a punt.