You can do this from the Configuration > Device Management > Management Access > ASDM/HTTPS/Telnet/SSH tab. Just make sure your entries for ASDM/HTTPS only contain IP addreses for devices you want to be able to manage the firewall. Anyone not in that list will be blocked from accessing ASDM.
It's a bit more involved, but you could setup a remote access VPN and setup ASDM to only allow external access to the IPs in the VPN pool. This way, you can provide the remote access credentials only to the administrators who should access the firewall.
Other than that, your best bet would be to do what Collin suggested and setup different user accounts with different privilege levels.
We have configured the outside and inside Interface with official ipv6 adresses, set a default route on outside Interface to our router, we also have definied a rule , which also gets hits, to permit tcp from inside Interface to any6.
In Syslog I also se...