Good question, and while there are certainly technical reasons for choosing one over the other, this discussion is occasionally as much political as it is technical :-)
All kidding aside, there are many factors to consider with such environments, and certainly in some cases, GigE or 10GigE will make sense, and in others, IB will have the edge. Rather then trying to post the many possibilities in this forum, there are some useful links that may help answer your question.
Lastly, if you want to get a feel for what others in the HPC world are doing (could be useful to see trends in technologies deployed), check out the top500 site, where you can do various searches on how many of the top 500 supercomputers in the world use what technology as an interconnect (as well as look at past lists to see the general direction these systems are taking with their interconnects):
Introduction This article will help you understand the steps on how to
download the UCS licenses from the Cisco Systems website and then
installing it on the UCS. The redacted (blue lines) just covers up
certain numbers for privacy please do not take them...
Introduction This article will help you understand and educate the
customer on how to clear their "expired licenses"
(license-graceperiod-expired) from their UCS-M. If a customer just
purchased a license and needs a step by step guide on how to download
==================== VIC FNIC driver does not support Virtual Volumes (
second level LUN ID ) An enhancement request has been created to track
this feature - CSCux64473 UPDATE - 12-14-2016 We made some traction on
the enhancement request - The Fix is in t...