ACE 30, dropped conns counter incorrect number

[p.hruby]

We have host in our network which tests reachability of ACE's VIP address at regular intervals. The test sequence consists of 4 TCP packets (SYN, SYN-ACK, FIN-ACK, RST-ACK; see picture attached) and causes incrementation of "dropped conns" counter in show service-policy output.

ACE30# sh service-policy XYZ detail | inc drop

        dropped conns    : 266812

        conn-rate-limit      : 0         , drop-count : 0

        bandwidth-rate-limit : 0         , drop-count : 0

                     dropped conns: 238177

            dropped conns    : 7

ACE30# sh service-policy XYZ detail | inc drop

        dropped conns    : 266813

        conn-rate-limit      : 0         , drop-count : 0

        bandwidth-rate-limit : 0         , drop-count : 0

                     dropped conns: 238178

            dropped conns    : 7

Is this normal behavior of ACE? Is there a way how to get rid of the dropped cons counter incrementation.

Petr

[Kanwaljeet Singh]

Hi Petr,

You should see why the connections are getting dropped. Is it due to some class-map condition not met or something else. The best way would be to take pcaps and see why ACE is dropping the connection. There is no other way to get rid of these dropped connections. The dropped connections are those which you didn't want ACE to allow at first place.

Regards,

Kanwal

[p.hruby]

Hi Kanwal,

In L3,L4 policy following class is configured:

class-map match-any VIP-ST
  2 match virtual-address 10.105.137.171 tcp eq https

In L7 policy class class-default is configured only.

I have taken pcaps using NAM module. The results are attached. As you can see no data are sent from client to VIP of ACE. The communication consists of control packets (SYN,FIN,RST) only.

It seems to me that ACE cannot manage those kind of test connections and increments dropped connections counter.

Kind regards

Petr

[Kanwaljeet Singh]

Hi Petr,

Something is not right here. If you see TCP 3-way handshake is not completing and client is sending a FIN ACK instead of ACK. There are retransmissions from both sides untill ACE sends the RST. ACE is expecting ACK for 3-way handshake to complete but instead gets FIN ACK.

In L4 load balancing, ACE just forwards the packet from front end to backend. So we shall see this FIN ACK at the backend server and if we will look at the backend packet capture the RST would be from server too. Now, this behavior might change if we have SSL termination since ACE acts as a proxy while doing SSL termination (behavior like L7 LB) and backend connection is not opened unless we get HTTP GET at the front end. While in L7 we need to do LB on the basis of info in http header, in ssl termination we may not so i am not very sure if we need to wait for HTTP GET or just open a backend connection as soon as we get SYN at the front end.

But if in your case it is just L4 LB(classification on TCP PORT 443), to TS further i would request to get a simultaneous pcaps so that we can see both front end as well as back end communication i.e from client to ace and ace to server and see what is going on.

But in any case client has no business to send FIN ACK without completing 3 way handshake.

Regards,

Kanwal

[p.hruby]

Hi Kanwal,

it seems to me that ACE is unable to handle those SYN,SYN-ACK,FIN sequences and expects (as you've mentioned) usual 3-way handshacking

I had thought that the sequence SYN,SYN-ACK,FIN is incorrect but then I found  TCP state-machine graph which describes this sequence as valid.

I agree it is strange when client finishes connection before it is completly established. But in this case the connection is created by host which only tests reachability of  VIP. In this case such behavior makes sense in my opinion.

You can see graph of ACE to real server communication in vip-droped-con.gif. I also attach pcap screenshot from Wireshark (All packets are duplicated in this graph. This is caused by NAM. In real communication packets are not duplicated).

Regards

Petr

[Kanwaljeet Singh]

Hi Petr,

I would be interested to see the behavior when you have "Normalization" disabled. Can you do "no normalization" and test again?

http://docwiki.cisco.com/wiki/Cisco_Application_Control_Engine_%28ACE%29_Troubleshooting_Guide_--_Troubleshooting_Connectivity

Look at this link and look at section:

TCP L4 connection set up with Normalization enabled.

If you see the connection only turns to ESTAB when ACK is seen from the client. Otherwise IN is SYN SEEN and OUT is SYNACK. And in our case ACK is never received. This is like embryonic connection. And ACE will send RST to both front end as well as back end to clear the connection. This is because of security feature and is due to NORMALIZATION.

Configuring the Timeout for an Embryonic Connection

Occasionally, the TCP three-way handshake for a connection may not complete for some reason. This type of connection is called an embryonic connection. To configure a timeout for embryonic connections, use the set tcp timeout embryonic command in parameter map connection configuration mode. The syntax of this command is as follows:

set tcp timeout embryonic seconds

The seconds argument is an integer from 0 to 4294967295 seconds. The default is 5 seconds. A value of 0 specifies that the ACE does not time out an embryonic connection.

Hope this helps.

Regards,

Kanwal

Note: Please mark the answers if they helped.

[p.hruby]

Hi Kanwal,

When I set "no normalization" problem is solved. Disadvantage of this appoach is that by this command all trafic on interface is affected.

I've also tried to tune  timeout for embrionic connection.

When I had set it to 0, dropped conns counter stopped to increase. Client which sends those "SYN,FIN" packets ends communication after 30 seconds using RST. This cause that connection ends and dropped conns counter does not increase.

Unfortunately for some reason sometimes happens that client doesn't send this final RST packet. This cause that number of active connection increases ...

ACE30-hto2/TEST-WEBAPP# sh service-policy XYZ | inc conn

        curr conns       : 9         , hit count        : 2279841   
        dropped conns    : 385467    
        conns per second    : 0         
        conn-rate-limit      : -         , drop-count : -         

ACE30-hto2/TEST-WEBAPP# sh service-policy XYZ | inc conn

        curr conns       : 22        , hit count        : 2283653   
        dropped conns    : 385467    
        conns per second    : 0         
        conn-rate-limit      : -         , drop-count : -

When I set timeout to 120, those "non RST" connections are cleared but of course dropped conns counter increases ...

I guess I will try to reconfigure the probe.

Kanwal, thanks for your suggestions!

Kind regards

Petr