Enable "debug tacacs" on the switch showing the problem, and see if enabling single connect causes the switch to connect to the first TACACS+ server, timeout, then go to the next one.
The default timeout value is 5 seconds, which is the delay that you are seeing, which is why I suspect the above.