cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
3740
Views
0
Helpful
8
Replies

flow control on Cisco Nexus fabric extenders

LA-Engineer
Level 1
Level 1

Cisco Nexus Fabric Extender host ports are configured for 'flow-control send' by default.

This is understandable since the fabric extender fabric interfaces are oversubscribed.  However, I just wanted to know under what condition will the fabric extender send tx-pauses?  

For instance, will the fabric extender send tx-pauses when the fabric interfaces are at a certain percentage utilized?

Thank you in advance!

 

1 Accepted Solution

Accepted Solutions

Reza Sharifi
Hall of Fame
Hall of Fame

What we have seen with 2232 series FEXs connecting to VM servers is that the unlink ports on the FEX send tx-pause frames when there is a pretty large transfer between ports. One of the problems with 2232 series is that all 8 uplink ports have only 12Mb shared buffer and so It can run out of buffer when there is a large transfer.  We have tried with ports having flow-control send off and on and also receive on and off and we are still seen pause frames.

What type of FEX are you using?

Are they connected to VM?

HTH

 

View solution in original post

8 Replies 8

Reza Sharifi
Hall of Fame
Hall of Fame

What we have seen with 2232 series FEXs connecting to VM servers is that the unlink ports on the FEX send tx-pause frames when there is a pretty large transfer between ports. One of the problems with 2232 series is that all 8 uplink ports have only 12Mb shared buffer and so It can run out of buffer when there is a large transfer.  We have tried with ports having flow-control send off and on and also receive on and off and we are still seen pause frames.

What type of FEX are you using?

Are they connected to VM?

HTH

 

Hi Reza, thanks for getting back to me.

We're are actually using a N2K-C2232PP-10GE!

The host ports are connected to a NetApp and are carrying iSCSI.

One other thing I noticed is that the host ports connected to the NetApp that are showing large numbers of Tx pauses are also recording output errors.  Its only one every few minutes but I'm wondering if this may be related.  I don't see any CRC errors anywhere to explain the output errors.

The reason I'm asking about this is because we are having performance issues on one of our servers that uses the NetApp connected to this FEX.  However, BW utilization on the fabric uplink port-channel doesn't really go above 250Mbps.  Perhaps the load interval isn't catching the spikes caused by iSCSI block requests?

Hi,

We don't see any output errors on the FEX uplinks.  Regarding performance issue, we also did some testing and noticed we don't get line rate performance on the 10Gig FEX interfaces. Since we have a ticket open with Cisco, they recommend to add "no hardware n2232p queue-limit" on the FEX uplink ports. After deploying this command, we are geeing much better performance (almost line rate) on the FEXs.  BTW, deploying this command does not require an outage and can be done any time.

I am interested to know to if this command will help you with NetApp performance issue.

HTH

 

 

Thank you.  I will setup a time to apply this and test whether or not it worked.

BTW, is there a way to see the status of these buffers such as the allocated size and whether there are drops?  Is it just "show queueing interface".

I think, in order to see the buffer parameters, you have to logging to the FEX.  From the switch, I think the command is "attach fex xxx" and then look for commands

HTH

Thank you.  I will try it this evening and let you know if there were any improvements.  By the way, would you happen to know why there is a limitation on the queue?  Why would anyone want to set the queue-limit?

Nevermind, I figured out the answer to that question.

Removing the queue-limit will allow a port to use up all of the shared buffer and blocking it for any other port.

I talked to TAC and they recommended the same thing - to remove the queue-limit on the host ports.  It looks like it made the performance of our SQL servers worse.  Although, our Netapp admin tells me that he's seeing fewer network related alarms.  Very strange.

I am going to move the netapp off the FEX and onto the parent switch.

Thanks for you help!

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community:

Innovations in Cisco Full Stack Observability - A new webinar from Cisco