We are experiencing a very strange behaviour on several CSS's in our webfarm environments.
We're running on a CSS 11506 with WebNS 8.10.1.07s.
The flow is very basic and simple:
1. A client connects to a VIP on our CSS.
2. Based on a content rule decision, the CSS forwards the packet to the real server IP
3. The server reply gets back through the load-balancer and the source-IP of the packet gets replaced by the VIP (the original IP the client had connected to)
We have some network traces and firewall logs showing that this last step (step 3 above) is sometimes missed for some packets (it works 99% of the time), on existing flows. Sometimes the CSS does NOT set the VIP as the source IP in the reply packets, but rather keeps the real server IP which obviously breaks the TCP paradigm/connection.
The flow is not being torn down, the wrong packet is just discarded by our firewall and, thanks to TCP, the reply is anyhow being retransmited later, and this time, properly handled by the CSS.
The configuration is OK (well, it's just failing on few packets for an existing and established flow) and it really looks like a bug in flow handling.
The CSS is not overloaded: I checked the CPU, the free memory as well as the free FCB left and we're far from the limit.
Any known issue ? Any idea of what to look at now to investigate further ?