flow control is the process of managing the rate of data transmission between two nodes to prevent a fast sender from outrunning a slow receiver. It provides a mechanism for the receiver to control the transmission speed, so that the receiving node is not overwhelmed with data from transmitting node.
If your device have flowcontrol enabled, and you disable it on the switch, than you can run into a mismatch that can cause your slow throughput rate. Depending on the device connected in the switch you have to enable flowcontrol (in your case this is true) or disable it. On your Cisco switch you can also configure receive desired and send desired...so only if the device connected there need flowcontrol, Cisco switch will enable it.
It might be that one device is sending at a rate higher than the other device can accept, and then you have this behavior with flowcontrol off. Flowcontrol help those devices realize that they can "push" more data than actually they would. I'm not expert with servers, but I believe this behavior is caused by the server side, as if you connect one router and one device together without any software limitation, they will send/receive at maximum rate.
It makes sense to me because only some type of servers (NAS) are experiencing this issue. Probably flow-control improves the performance for FCP communications. I am chasing the server team to provide me more details about the servers applications and protocols.
Btw, are you using Jumbo frames (MTU 9216)? If yes, have you configured them on both sides switch and server (if supported). If you have a mismatch there, you can run into low throughput due to fragmentation. Just and idea that might help.
Jumbo frames are not being used. However the problem resides in the servers NIC itself. The problem was a misconfiguration on the NIC parameter Interrupt Moderation which needs to be taken in count for NAS traffic flows due to the high amount of data.
We are pleased to announce availability of Beta software for 16.6.3. 16.6.3 will be the second rebuild on the 16.6 release train targeted towards Catalyst 9500/9400/9300/3850/3650 switching platforms. We are looking for early feedback from custome...