Which VPN client are you using?
There are some inbound ports you will need to open up for vpn connectivity.
For Cisco's 3.X client, try udp port 500, and IP port esp. Also you may need to open either udp port 10000, or udp port 4500. Check your syslog to see which packets are trying to get back in to verify this.
For Windows VPN client try opening IP GRE.
Some clients also require IP AH.
Hope that helps
~rls