Dave,
I would suggest to do some troubleshooting before jumping to conclusion.
The number of servers does not matter.
What is important is the number of connections.
1 connection per server means 20 ports being used and you have 55k available.
So, in this example this is definitely not a problem.
Most of the times, client drops on a CSS are due to the aggressive idle timeout. 16 sec.
So, either you try to bump the timeout with the command 'flow-timeout-multiplier' or you capture sniffer traces showing the drops with couple of show tech and we try to confirm the cause of the problem.
Gilles.