You might want to check the load that those VPN sessions put on the box as well as the CPU utlization to see if either value is excessive.
The Monitor section provides extensive statistics which may also help isolate a load issue.
The Filterable Event Log may provide some insight into system and VPN activities to help pinpoint a problem.
One thing I've found that adds a lot of overhead is excessive logging. Check the number of classes you are logging and the severity levels that you have set.
HTH