No you will be more than fine!
Many people have far less bandwidth and far more users.
What you can do is implement WFQ via MQC so you will provide fair queuing. You use
"shape average" statement to plug in what bandwidth you want the qos to go by. Say your interface is 100Mbps to the cable modem (or even 10Mbps) You simply tell WFQ the outbound and inbound bandwidth.
WFQ will prevent one guy doing a big download from slowing everyone else down. I would also look into dropping junk traffic early at the router.
shoot me an email and I'll send you the qos configs..
joe@affirmedsystems.com