We are facing an issue with a cisco CSM Load Balancer Module running version 3.1(4).
The scenario is that we have two IBM WebSphere Application Servers that we are trying to load balance using CSM. During perfromance testing, the systems team tried to simulate 50 simulaneous users to the Load Balanced VIP IP (10.10.10.1) on port 443. When both real servers are UP and running, we saw the connections were equally distributed among the two real servers (10.10.10.2 and .3).
However when they shut down one of the real servers NIC, we see that CSM marks the server as operationally down (displaying PROBE_FAILED status). But as soon as one of the real server fails, the CSM stops all requests for the virtual server, and the client application shows that no requests are reaching any of the real servers for about 10 seconds. After this period normal connections are established with ONE Active Real Server. We thought that if one real server goes, at least half of the connections should still continue to work.
Please see attached graph for more information. What could be the reason for this issue? Below is the relevant configuration:
probe ICMP icmp
no nat client
virtual 10.10.10.1 tcp https
no persistent rebalance