There are several quality of service options you could use to limit the number of connections to a host. There are a few factors that would make one choice better than the others (ie. type of traffic, platforms, etc).
Here is a url with information on weighted random early detection
