After upgrading ACE module from A2(3.1) to A2(3.2) we started getting complaints about the ablity to upload a file in one of our applications. We are able to recreate the problem and noticed that when there is a failure, the ACE sends a TCP RST in both directions and closes the connection with the Exception reason. This is in the middle of a prefectly good conversation and there does not appear to be an external reason for it. I noticed a couple of other discussions with a similar problem but with no conclusions. Looking at the caveats in the release notes does not give any clues to this being a known bug. Has anyone else dealt with this problem and found a fix?
Here is some information to illustrate.
Sep 16 2010 13:48:27 Core-FWSM : %FWSM-6-302013: Built outbound TCP connection 145057001608450058 for inside:10.3.66.209/3605 (10.3.66.209/3605) to Burnet_dmz:10.2.0.56/443 (10.2.0.56/443)
Sep 16 2010 13:48:26 DMZ: %ACE-6-302022: Built TCP connection 0x21280c for vlan120:10.3.66.209/3605 (10.3.66.209/3605) to vlan130:10.2.0.56/443 (10.2.0.151/443)
Error Message %ACE-6-302023: Teardown TCP connection id for interface:real-address/real-port to interface:real-address/real-port duration hh:mm:ss bytes bytes [reason]
Explanation This informational message is logged when a TCP connection slot between two hosts is terminated. The reason variable presents the action that causes the connection to terminate. Table 2-1 lists the TCP termination causes.
Table 2-1 TCP Termination Reasons
Reason Description TCP FINs Normal close down sequence. TCP Reset A TCP reset is received. Idle Timeout TCP connection is timed out. FIN Timeout TCP FIN timeout. SYN Timeout TCP SYN timeout. Exception Connection setup error. Policy Close A policy closes the TCP connection. Voluntary Close TCP connection is closed voluntarily by a user. Rebalance HTTP rebalance. Reuse Conn. Connection is reused. Reap Conn. Connection is closed due to control plane reap messages. Xlate clear Connection is closed due to execution of a clear xlate command. Conn clear Connection is closed due to execution of a clear conn command.
You are likely running into the following defect which only exists in A2(3.2)
CSCti88248 ACE resets TCP connection without waiting for reassembly timeout.
We would need a trace to confirm if it is a match or not, so I would suggest you open a tac case and provide us a trace of the failure or you can apply the workaround mentioned in the bug details to see if it resolves the issue.
TAC Customer Support Engineer
CCIE R&S - 36768
Engineer, Customer Support
Thanks for the reply Joel. You are correct we were experincing the noted bug. We opened a TAC case and worked with Jim Sirstin to identify that this was our problem. We implemented the suggested work around and it solved the problem. Here is the detail for the bug, which is in the ACE release notes.
ACE resets TCP based client connection in case there is packet loss from client end and ACE is waiting to re-assemble client traffic.
In an environment where:
(1) ACE is configured with a L7 load-balance policy where ACE proxies the client side TCP connection before making a load-balancing decision,
(2) Client side connection experiences packet loss and
(3) "TCP TX racing messages (data) " counter from ACE CLI command "show np X me-stats -stcp" output is incrementing.
Note: This problem can also occur with secure (SSL) terminated connections.
Configure an "empty" connection parameter-map and add it to multi-match policy-map under class-map configured for the VIP experiencing the problem.
Topology & Design:
Two ACI fabrics
Stretching VLANs using OTV
Both fabrics are advertising BD subnets into same routing domain
Some BDs(or say VLANs) are stretched, but some are not.
Endpoints can move betwee...
VMware Trunk Port Group is supported from ACI version 2.1
VMM integration must be configured properly
ASA device package must be uploaded to APIC
ASAv version must be compatible with ACI and device package version
Topology &Design:Traffic flow within same fabric:Endpoint moves to Fabric-2Bounce Entry Times OutTraffic Black-holedSummarySolutionAppendix:
In the Previous articles of ACI Automation, we are using Postman/Newman a...