Ideally, you want QoS, outbound on both ends, but even begging your ISP isn't always effective to obtain QoS. (They usually will sell you more bandwidth, even when you can't buy from them QoS.)
There are two techniques that can be used to control, to some extent, non-VPN inbound TCP traffic from flooding your link.
One technique is to rate-limit, or police, the non-VPN inbound TCP traffic. TCP attempts to increase its speed during large transfers, but slows down when it detects packet drops.
Another tecnique is to shape outbound ACKs for the non-VPN inbound TCP traffic. TCP will only send so many packets until waiting for return ACKs.
For non-TCP traffic, the rate-limit techique might work, or it might not.
PS:
If a "cheap" business class ADSL or cable service is available, you might want to split off non-critical traffic to a separate Internet link. (If you're clever, you can use each link to provide failover for the other.)