Are you doing split tunnelling? If so the client will have to send traffic to a host in the network that you're trying to connect from BEFORE that network will be able to connect to the PC.
For example, if you're only tunnelling traffic for the 10.0.0.0/8 and 172.16.0.0/16 networks, no-one on either of these networks will be able to initiate a connection to the client machine, until that client machine has sent traffic to those particular networks. Keep in mind that an IPSec tunnel (SA) is not actually built until traffic is sent to that network.
Also check the "Stateful Firewall (Always On) option under the Options menu on the client, you might need to turn it off.