ACE does not terminate Half-closed connections after idle timeout expire

[cbm-it-iii]

Hello,

We use Csico ACE (ACE30-MOD-K9) for ssl-offload and loadbalancing between two real servers. In summary our ACE configuration is:

2 real servers

HTTPS protocol

Round-robin for predictor

sticky based on src/dst IP

persistent rebalance

parameter map with :

                inactivity timeout (seconds)       : 1800

                half-closed timeout (seconds)      : 1200

Unfortunately there is many Half-Closed connections that never expired. So my question is Why and What is happed?

I note the following ACE behavior, for example:

conn-id    np dir proto vlan source                destination           state

----------+--+---+-----+----+---------------------+---------------------+------+

635805     1  in  TCP   4    192.168.254.233:3572  172.16.3.201:443      ESTAB

          [ idle time   : 04:32:16,   byte count  : 4367       ]

          [ elapsed time: 04:32:43,   packet count: 17         ]

602267     1  out TCP   17   172.16.17.111:80      192.168.254.233:3768  CLOSED

          [ conn in reuse pool : FALSE]

          [ idle time   : 00:00:25,   byte count  : 11537      ]

          [ elapsed time: 04:32:43,   packet count: 146        ]

Connection 60227 is in CLOSE state but in every two minutes byte count is increased and idle timeout is reset. I thing that this is the cause which bring connection 635805 in ESTAB state despite of idle time exceed?! I made a capture and there are any traffic showed between

172.16.17.111:80   and   192.168.254.233:3768. When I set the half-closed timeout less than two minutes, all similar connections are terminated.

Is it a bug?

the ACE module  software is:

Version A4(1.0) [build 3.0(0)A4(1.0)

[Kanwaljeet Singh]

Hi,

Half closed connection is when server/client sends a FIN and client/Server ACK's the FIN but doesn't send it's own FIN.

In above connection it seems that server sent the FIN but client didn't. Now since every 2 minutes byte count increases there must be some communication. Where did you take the pcaps?  Do you see any packets ovcr the CLOSED connection? I will try to find out more details about this if no one has an answer by then:)

Regards,

Kanwal

[cbm-it-iii]

Hi Kanwal,

I did the following capture in the ACE, but there is no any traffic captured.

access-list BPA-access line 8 extended permit tcp host 172.16.17.111 eq www host 192.168.254.233 eq 3768

access-list BPA-access line 10 extended permit tcp host 192.168.254.233 eq 3768 host 172.16.17.111 eq www

capture BPAcap all access-list BPA-access

capture BPAcap start

Regards,

[cbm-it-iii]

Hi again, meanwhile i ask for a tcp dump from the one of the real servers which showed the traffic between server and ACE and I saw that: 

1. the server 172.16.17.111 sent [FIN, ACK]

2. the client 192.168.254.233 replay with [ACK]

3. server send again [FIN,ACK]

4. client replay with a duplicate ack [TCP dup ACK 2#1]

5. server again sent a [FIN,ACK]

6. client replay with dup ack [TCP dup ACK 2#2]

...... and so on

the problem might be the missing of [FIN,ACK] from the client toward the server. So is it a ACE or server problem?

[Kanwaljeet Singh]

Hi,

I think the problem is neither with server nor with ACE. The client needs to send a FIN for the ACE to completely close the connection. Now since server keeps sending packets (FIN ACK) it must be resetting the time out value and that's the reason why backend connection never reaches idle timeout. The below DDTS was introduced where client never replied to FIN even with ACK which is not the case in your case. You have client replying to FIN and connection moving to CLOSED state but i guess because of server sending FIN ACK again and again i resetting the idle time out and hence it is not going out of connection table.

CSCtr61749:  ACE equivalent of CSM CSM_FAST_FIN_TIMEOUT functionality

I would open a TAC case to investigate this further since it may require inputs from development.

Regards,

Kanwal