You can check the "group lock" feature in the group settings.
It will lock different users into different group.
A normal user import admin group settings, but his username does not belong to admin group, so he can not pass the user authentication, he still can not get VPN connection.