Hello Sarenos,
It is hard to say what is causing the latency between your management host and switch.
"When I ping to the switches the pings are very large 60--100 ms"
Do you have more VLANs configured other than the default VLAN 1? In case you have host in a other VLAN, is your management VLAN by any chance different than the VLAN for your hosts and/or servers?
Routing between VLANs adds latency and - if configured wrong - could affect performance. The issue could also be the result of a poor ARP response from the switch.
Either way you have to search your topology for possible problems. Start by using a network sniffer and look for ARP requests and replies between your host and the switch.
"pings between all devices are correct <1ms"
This is obviously because the hosts reside on the same switch and/or same VLAN.
If you need further assistance please provide a diagram and configuration snippets.
HTH
Leon
* Please rate posts.