ACE Exception during conversation

Unanswered Question
Sep 16th, 2010

After upgrading ACE module from A2(3.1) to A2(3.2) we started getting complaints about the ablity to upload a file in one of our applications. We are able to recreate the problem and noticed that when there is a failure, the ACE sends a TCP RST in both directions and closes the connection with the Exception reason. This is in the middle of a prefectly good conversation and there does not appear to be an external reason for it. I noticed a couple of other discussions with a similar problem but with no conclusions. Looking at the caveats in the release notes does not give any clues to this being a known bug. Has anyone else dealt with this problem and found a fix?

Here is some information to illustrate.

Sep 16 2010 13:48:27 Core-FWSM : %FWSM-6-302013: Built outbound TCP connection 145057001608450058 for inside:10.3.66.209/3605 (10.3.66.209/3605) to Burnet_dmz:10.2.0.56/443 (10.2.0.56/443)

Sep 16 2010 13:48:26 DMZ: %ACE-6-302022: Built TCP connection 0x21280c for vlan120:10.3.66.209/3605 (10.3.66.209/3605) to vlan130:10.2.0.56/443 (10.2.0.151/443)

Sep 16 2010 13:49:22 DMZ: %ACE-6-302023: Teardown TCP connection 0x21280c for vlan120:10.3.66.209/3605 (10.3.66.209/3605) to vlan130:10.2.0.56/443 (10.2.0.151/443) duration 0:00:55 bytes 2554818 Exception

Sep 16 2010 13:49:22 Core-FWSM : %FWSM-6-302014: Teardown TCP connection 145057001608450058 for inside:10.3.66.209/3605 to Burnet_dmz:10.2.0.56/443 duration 0:00:55 bytes 2641677 TCP Reset-O


ACE error message 302023

Error Message    %ACE-6-302023: Teardown TCP connection id for interface:real-address/real-port to interface:real-address/real-port duration hh:mm:ss bytes bytes [reason]

Explanation    This informational message is logged when a TCP connection slot between two hosts is terminated.
The reason variable presents the action that causes the connection to terminate. Table 2-1 lists the TCP termination causes.

Table 2-1 TCP Termination Reasons

Reason                Description
TCP FINs             Normal close down sequence.
TCP Reset           A TCP reset is received.
Idle Timeout         TCP connection is timed out.
FIN Timeout         TCP FIN timeout.
SYN Timeout       TCP SYN timeout.
Exception            Connection setup error.
Policy Close        A policy closes the TCP connection.
Voluntary Close   TCP connection is closed voluntarily by a user.
Rebalance           HTTP rebalance.
Reuse Conn.        Connection is reused.
Reap Conn.          Connection is closed due to control plane reap messages.
Xlate clear           Connection is closed due to execution of a clear xlate command.
Conn clear            Connection is closed due to execution of a clear conn command.

Recommended Action    None required.

I have this problem too.
0 votes
  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 0 (0 ratings)
Loading.
jlamousn Fri, 09/17/2010 - 08:21

You are likely running into the following defect which only exists in A2(3.2)


CSCti88248    ACE resets TCP connection without waiting for reassembly timeout.

We would need a trace to confirm if it is a match or not, so I would suggest you open a tac case and provide us a trace of the failure or you can apply the workaround mentioned in the bug details to see if it resolves the issue.

Thanks

Joel Lamousnery

TAC Customer Support Engineer

klogan Mon, 09/20/2010 - 15:48

Thanks for the reply Joel. You are correct we were experincing the noted bug. We opened a TAC case and worked with Jim Sirstin to identify that this was our problem. We implemented the suggested work around and it solved the problem. Here is the detail for the bug, which is in the ACE release notes.

Symptom:

ACE resets TCP based client connection in case there is packet loss from client end and ACE is waiting to re-assemble client traffic.

Conditions:

In an environment where:

(1) ACE is configured with a L7 load-balance policy where ACE proxies the client side TCP connection before making a load-balancing decision,

(2) Client side connection experiences packet loss and

(3) "TCP TX racing messages (data) " counter from ACE CLI command  "show np X me-stats -stcp" output is incrementing.

Note: This problem can also occur with secure (SSL) terminated connections.

Workaround:

Configure an "empty" connection parameter-map and add it to multi-match policy-map under class-map configured for the VIP experiencing the problem.

Example:

parameter-map type connection TCPReassembly

  policy-map multi-match MultiMatch_PolicyMap

       class HTTP_VIP_80

          loadbalance vip inservice

          loadbalance policy L7_HTTP_PolicyMap

          loadbalance vip icmp-reply active

          connection advanced-options TCPReassembly

Actions

This Discussion