Latency isnt causing your main problem here, I've just pinged a remote site with a similar latency:
Sending 5, 100-byte ICMP Echos to 172.16.65.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 492/497/504 ms
There are a couple of common problems I could think of that may be causing this failure. The first is that your crossing a NAT boundary somewhere, in which case the VPN does authenticate as it uses ISAKMP over UDP 500, but the IPSec encapsulated data fails due to a NAT altering the packets. Enabling NAT-T support fixes this.
The other is down to routing, one end not knowing how to route to the remote via the VPN link.