Half-Width Blade's Bandwidth and IOM

Answered Question
May 31st, 2011

Hi, Cisco Gurus:

I have read that Cisco B-Series Half-Width Blade has an aggregate bandwidth of 20GB, may I seek some authoritative answers from this forum?

Example:

2 IOM/2 FInterconnect with 8 Half-Width Blades in UCS 5108 Chassis-2 Server Links Per IOM-Left and another 2 Server Links on Right IOM

Q1.

With the above setup, which Fabric or IOM will be used, A or B?

                            Static Pinning

Server Link 1         Slot/Blade 1,3,5,7---------------- Which IOM will be used for these Blades?

Server Link 2         Slot/Blade 2,4,6,8---------------- Which IOM will be used for these Blades?

Q2.

How to derive the 20G bandwidth in Cisco documentation?

Q3.

If Blade 1 O/S is doing a write I/O request, say deterministically via IOM A (Left of Chassis), if there is another Read I/O issued by Blade 1 again, will the I/O path go through IOM B(Right of Chassis) or will it go through IOM A again? Is there any algorithm here in determining this?

Q4.

How to design a solution which will load-balance between the 2 IOM? Or we can leave it to Cisco UCSM to decide automatically?

I undertsand that I can check Enable Failover using UCSManager to manullay select either Fabric A or Fabric B as the Primary Path and the other as Secondary in A-B (A is the Pri) or B-A (B as the Pri).

Please help me to help Cisco sell more UCS as the above is the burining questions of some of us.

Thanks.

Sim

I have this problem too.
0 votes
Correct Answer by roberbur about 2 years 10 months ago

Inline.

ciscoucsisit wrote:

Rob:

Getting better understanding now. Excellent help from you.

Assuming I create 1 service profile with the following:

Adapter 1 as seen within UCSM

vHBA-A- Fabric A - Is this 4GB as of now or can we treat it as 10G Port too likes vNIC?

Depends on the Mezz card.  M71/M81 have 4GB FC chipsets.  M72KR uses an 8GB FC chip.

vHBA-B- Fabric B - Same as above

vNIC-A-Fabric A-This is 10G port, for sure, right.

Yes

vNIC-B-Fabric B-Same as above

For VMware vDS setup, say for vMotion/Service Console to have Active and Failover/Redundant vmnic, should I team the 2 vNIC-A and vNIC-B or

I should CREATE another Adapter and team the 2 ports or 1 port here with Adapter 1?

Depending on which adapter you're using, you can create max to 2 NICs (M71KR/M72KR) or up to 56 NICs (M81KR).  If you can only create a max of 2 NICs you'll need to use both NICs as uplinks to your vDS - both will be active. With Palo (M81KR)

It is creating a lot of confusion for non-Cisco Blade folks even from VMware and me too here as "physically" it is still only 1 M81KR with Dual-Ports for vHBA and vNIC alike???

The value add for the M81 (Palo) is you're able to create multiple virtual NICs/HBAs which can all be treated individually (VLANs, QoS, Security, Stat Counters etc)You are correct that you're still limited to two phsyical underlying NICs on each blade - where your design questions come into play is how you want to manage the behavior & performance of each NIC.

Can you please throw some light into it as I think it is all in your head?

You have done a great job in enlightening me and the rest.

Sim

Regards,

Robert

  • 1
  • 2
  • 3
  • 4
  • 5
Average Rating: 5 (3 ratings)
roberbur Tue, 05/31/2011 - 18:09

Sim,

Correct.  Each 1/2 width slot can access a 10G/FCoE connection to each IO Module.  Current IO modules support 4 x 10G FCoE connections upstream to one Fabric Interconnect respectfully.  Oversubscription between a Chassis & Fabric Interconnect will vary from 2:1, 4:1 to 8:1 depending on how many Server links are connected & acknowledged (1,2, or 4).

Rest of answers are inline.

ciscoucsisit wrote:

Hi, Cisco Gurus:

I have read that Cisco B-Series Half-Width Blade has an aggregate bandwidth of 20GB, may I seek some authoritative answers from this forum?

Example:

2 IOM/2 FInterconnect with 8 Half-Width Blades in UCS 5108 Chassis-2 Server Links Per IOM-Left and another 2 Server Links on Right IOM

Q1.

With the above setup, which Fabric or IOM will be used, A or B?

                            Static Pinning

Server Link 1         Slot/Blade 1,3,5,7---------------- Which IOM will be used for these Blades?

Server Link 2         Slot/Blade 2,4,6,8---------------- Which IOM will be used for these Blades?

Each blade has a path to each IOM. When defining vNICs within UCS, you define whether the host NIC connects to Fabric A, Fabric B or both (Fabric Failover).  Depending how you cabled each IOM, will determine which path is taken.  Normally IOM1 (Left) goes to FI A and IOM 2 (Right) connects to FI B.


 

Q2.

How to derive the 20G bandwidth in Cisco documentation?

The ports on all Mezz cards currently available at 10G Interfaces.  Each Mezz card has two ports.  Each 1/2 width blade supports 1 Mezz card (Full Width support 2 Mezz cards @ 40G). Both ports are Active, there is no Active/Standby behavior of these two interfaces.

Q3.

If Blade 1 O/S is doing a write I/O request, say deterministically via IOM A (Left of Chassis), if there is another Read I/O issued by Blade 1 again, will the I/O path go through IOM B(Right of Chassis) or will it go through IOM A again? Is there any algorithm here in determining this?

Path selection is an OS-level function.  For VMware this will depend on your vSwitch hashing, for RedHat your Bonding Type, for Windows it's Teaming.  VMware & Redhat have built-in support for teaming/multipathing, where Windows requires vendor-specific drivers.  Emulex & Qlogic provide these packages for Windows for the M71KR & M72KR series of UCS adapters.  Similar packages available from Intel & Broadcom for Windows.

Q4.

How to design a solution which will load-balance between the 2 IOM? Or we can leave it to Cisco UCSM to decide automatically?

I undertsand that I can check Enable Failover using UCSManager to manullay select either Fabric A or Fabric B as the Primary Path and the other as Secondary in A-B (A is the Pri) or B-A (B as the Pri).

Please help me to help Cisco sell more UCS as the above is the burining questions of some of us.

Remember UCS does not "balance" traffic dynamically between IOMs.  Each host's NIC or Driver will handle this function.  In regards to Fabric Failover there are two adapters that can support this - M71KR (Menlo) & M81KR (Palo).  Since both the Menlo & Palo mezz card chips were created BY Cisco, we have control over which path traffic between the adapter & IOM traverses. When you enable Fabric Failover for each Interface a secondary Interface on the opposite Fabric is instantiated.  What this provides is seemless failover in the event the path between Adapter & Fabric Interconnect fails or goes down.  The traffic at the adapter level is redirected out the functional port, through the IOM and up to the Fabric Interconnect.  Though this does cause traffic to use the same link, the host will not detect any failures at the networking level. 


For more information I suggest you have a look through some of the videos Brad Hedlund has prepared here:

http://bradhedlund.com/2011/03/08/cisco-ucs-networking-videos-in-hd-updated-improved/

Let me know if you have any other specific questions.

Regards,

Robert

ciscoucsisit Tue, 05/31/2011 - 18:44

roberbur wrote:

Sim,

Correct.  Each 1/2 width slot can access a 10G/FCoE connection to each IO Module.  Current IO modules support 4 x 10G FCoE connections upstream to one Fabric Interconnect respectfully.  Oversubscription between a Chassis & Fabric Interconnect will vary from 2:1, 4:1 to 8:1 depending on how many Server links are connected & acknowledged (1,2, or 4).

Rest of answers are inline.

ciscoucsisit wrote:

Hi, Cisco Gurus:

I have read that Cisco B-Series Half-Width Blade has an aggregate bandwidth of 20GB, may I seek some authoritative answers from this forum?

Example:

2 IOM/2 FInterconnect with 8 Half-Width Blades in UCS 5108 Chassis-2 Server Links Per IOM-Left and another 2 Server Links on Right IOM

Q1.

With the above setup, which Fabric or IOM will be used, A or B?

                            Static Pinning

Server Link 1         Slot/Blade 1,3,5,7---------------- Which IOM will be used for these Blades?

Server Link 2         Slot/Blade 2,4,6,8---------------- Which IOM will be used for these Blades?

Each blade has a path to each IOM. When defining vNICs within UCS, you define whether the host NIC connects to Fabric A, Fabric B or both (Fabric Failover).  Depending how you cabled each IOM, will determine which path is taken.  Normally IOM1 (Left) goes to FI A and IOM 2 (Right) connects to FI B.


 

Q2.

How to derive the 20G bandwidth in Cisco documentation?

The ports on all Mezz cards currently available at 10G Interfaces.  Each Mezz card has two ports.  Each 1/2 width blade supports 1 Mezz card (Full Width support 2 Mezz cards @ 40G). Both ports are Active, there is no Active/Standby behavior of these two interfaces.

Q3.

If Blade 1 O/S is doing a write I/O request, say deterministically via IOM A (Left of Chassis), if there is another Read I/O issued by Blade 1 again, will the I/O path go through IOM B(Right of Chassis) or will it go through IOM A again? Is there any algorithm here in determining this?

Path selection is an OS-level function.  For VMware this will depend on your vSwitch hashing, for RedHat your Bonding Type, for Windows it's Teaming.  VMware & Redhat have built-in support for teaming/multipathing, where Windows requires vendor-specific drivers.  Emulex & Qlogic provide these packages for Windows for the M71KR & M72KR series of UCS adapters.  Similar packages available from Intel & Broadcom for Windows.

Q4.

How to design a solution which will load-balance between the 2 IOM? Or we can leave it to Cisco UCSM to decide automatically?

I undertsand that I can check Enable Failover using UCSManager to manullay select either Fabric A or Fabric B as the Primary Path and the other as Secondary in A-B (A is the Pri) or B-A (B as the Pri).

Please help me to help Cisco sell more UCS as the above is the burining questions of some of us.

Remember UCS does not "balance" traffic dynamically between IOMs.  Each host's NIC or Driver will handle this function.  In regards to Fabric Failover there are two adapters that can support this - M71KR (Menlo) & M81KR (Palo).  Since both the Menlo & Palo mezz card chips were created BY Cisco, we have control over which path traffic between the adapter & IOM traverses. When you enable Fabric Failover for each Interface a secondary Interface on the opposite Fabric is instantiated.  What this provides is seemless failover in the event the path between Adapter & Fabric Interconnect fails or goes down.  The traffic at the adapter level is redirected out the functional port, through the IOM and up to the Fabric Interconnect.  Though this does cause traffic to use the same link, the host will not detect any failures at the networking level. 


For more information I suggest you have a look through some of the videos Brad Hedlund has prepared here:

http://bradhedlund.com/2011/03/08/cisco-ucs-networking-videos-in-hd-updated-improved/

Let me know if you have any other specific questions.

Regards,

Robert


Rob:

Thank. Great, u make my day.

More clarifications:

Q1.

If I DO NOT select Enable Failover for the 2 Service Profiles for my 2 Blades and with the following definition at vNIC level, can I say that I am ONLY using Fabric A UNTIL FAILOVER?

Blade 1's Service Profile with 2 vNICs-vNIC-A associated with Fabric A, vNIC-B with Fabric B (without Enable Failover)

Blade 2's ----------------------- The SAME as above

Q1.1

If I DO NOT select Enable Failover, what will happen during ACTUAL FAILURE of Fabric A assuming using the same Service Profiles as Q1 above?

Will UCSM route the traffic to Fabric B even WITHOUT selecting Enable Failover using UCSM?

Q3.

Where can I have more info for vSwitch Hashing? Is it something I can change/set or is it by default?

roberbur Tue, 05/31/2011 - 19:11

Q1.

If  I DO NOT select Enable Failover for the 2 Service Profiles for my 2  Blades and with the following definition at vNIC level, can I say that I  am ONLY using Fabric A UNTIL FAILOVER?

Blade 1's Service Profile with 2 vNICs-vNIC-A associated with Fabric A, vNIC-B with Fabric B (without Enable Failover)

Blade 2's ----------------------- The SAME as above

No, if you do NOT select FF, then a failure of the Interconnect will take down corresponding vNIC on the host.  In this case as long as you're doing NIC teaming at the host level you shouldn't have any overall loss of network connectivity to your host/VMs.

Q1.1

If  I DO NOT select Enable Failover, what will happen during ACTUAL FAILURE  of Fabric A assuming using the same Service Profiles as Q1 above?

Will UCSM route the traffic to Fabric B even WITHOUT selecting Enable Failover using UCSM?

Same answer as above.

Q3.

Where can I have more info for vSwitch Hashing? Is it something I can change/set or is it by default?

By default vSwitch hashing is based on Virtual Port ID.  This means each VM will be pinned to one of the vSwitch's phsycial uplinks.  If one of the uplinks fails, traffic will be re-pinned to one of the remaining functional uplinks.  In regards to UCS, the default hashing method is the only one you should be using.  (Other options include "IP Hashing" which is used when the upstream uplinks are Port Channeled together - which is not possible with UCS at this time).


More information for vSwitch hashing and VMware NIC teaming here:

http://www.tcpdump.com/kb/virtualization/vmware-esx-server/esx-nic-teaming/intro.html

http://www.scribd.com/doc/50447337/27/vSwitch-and-NIC-Teaming-Best-Practices

http://blog.scottlowe.org/2008/07/16/understanding-nic-utilization-in-vmware-esx/

Regards,

Robert

ciscoucsisit Tue, 05/31/2011 - 20:41

Rob:

Getting better understanding now. Excellent help from you.

Assuming I create 1 service profile with the following:

Adapter 1 as seen within UCSM

vHBA-A- Fabric A - Is this 4GB as of now or can we treat it as 10G Port too likes vNIC?

vHBA-B- Fabric B - Same as above

vNIC-A-Fabric A-This is 10G port, for sure, right.

vNIC-B-Fabric B-Same as above

For VMware vDS setup, say for vMotion/Service Console to have Active and Failover/Redundant vmnic, should I team the 2 vNIC-A and vNIC-B or

I should CREATE another Adapter and team the 2 ports or 1 port here with Adapter 1?

It is creating a lot of confusion for non-Cisco Blade folks even from VMware and me too here as "physically" it is still only 1 M81KR with Dual-Ports for vHBA and vNIC alike???

Can you please throw some light into it as I think it is all in your head?

You have done a great job in enlightening me and the rest.

Sim

Correct Answer
roberbur Tue, 05/31/2011 - 21:47

Inline.

ciscoucsisit wrote:

Rob:

Getting better understanding now. Excellent help from you.

Assuming I create 1 service profile with the following:

Adapter 1 as seen within UCSM

vHBA-A- Fabric A - Is this 4GB as of now or can we treat it as 10G Port too likes vNIC?

Depends on the Mezz card.  M71/M81 have 4GB FC chipsets.  M72KR uses an 8GB FC chip.

vHBA-B- Fabric B - Same as above

vNIC-A-Fabric A-This is 10G port, for sure, right.

Yes

vNIC-B-Fabric B-Same as above

For VMware vDS setup, say for vMotion/Service Console to have Active and Failover/Redundant vmnic, should I team the 2 vNIC-A and vNIC-B or

I should CREATE another Adapter and team the 2 ports or 1 port here with Adapter 1?

Depending on which adapter you're using, you can create max to 2 NICs (M71KR/M72KR) or up to 56 NICs (M81KR).  If you can only create a max of 2 NICs you'll need to use both NICs as uplinks to your vDS - both will be active. With Palo (M81KR)

It is creating a lot of confusion for non-Cisco Blade folks even from VMware and me too here as "physically" it is still only 1 M81KR with Dual-Ports for vHBA and vNIC alike???

The value add for the M81 (Palo) is you're able to create multiple virtual NICs/HBAs which can all be treated individually (VLANs, QoS, Security, Stat Counters etc)You are correct that you're still limited to two phsyical underlying NICs on each blade - where your design questions come into play is how you want to manage the behavior & performance of each NIC.

Can you please throw some light into it as I think it is all in your head?

You have done a great job in enlightening me and the rest.

Sim

Regards,

Robert

ciscoucsisit Tue, 05/31/2011 - 22:32

Robert:

Again, appreciate it!

Any place I can rate you for a well-done job to further enable more customers to jump into Cisco UCS Blade camp!

More questions I will ask you, Robert.

Take care.

Sim

ciscoucsisit Wed, 06/15/2011 - 18:01

Robert:

After deeper maybe unnecessary thoughts, can you help to clarify the following?

q1. If I created 4 vNIC in my UCS Service Profile and I install VMware Hypervisor onto that same server and assuming that Service Console will use vmnic0, I know that I will see 4 vmnic as in VMware term.

vNIC-A-1 - Fabric A - vmnic0

vNIC-B-1 - Fabric B - vmnic1

vNIC-A-2 - Fabric A - vmnic2

vNIC-B-2 - Fabric B - vmnic3

Does I or VMware have to care about load-balancing among the 4 vmnic to achieve even utilization at all?

I understand that we can do NIC Teaming. Say in this case, I am using Active-Active NIC Teaming and Active-Passive NIC Teaming.

q2. It looks like there is ALWAYS 1-to-1 correspondence between what I  have created in UCS Service Profile and what vSphere had discovered, am I right?

Thanks, again.

SiM

roberbur Wed, 06/15/2011 - 19:31

Answers inline.

K P SIM wrote:

Robert:

After deeper maybe unnecessary thoughts, can you help to clarify the following?

q1. If I created 4 vNIC in my UCS Service Profile and I install VMware Hypervisor onto that same server and assuming that Service Console will use vmnic0, I know that I will see 4 vmnic as in VMware term.

vNIC-A-1 - Fabric A - vmnic0

vNIC-B-1 - Fabric B - vmnic1

vNIC-A-2 - Fabric A - vmnic2

vNIC-B-2 - Fabric B - vmnic3

Does I or VMware have to care about load-balancing among the 4 vmnic to achieve even utilization at all?

[Robert] The default hashing of a vSwitch is based on the Virtual Port ID - which means you can utilize multiple uplinks in Active/Active operation.  This will be a round-robin style of assigning a VM to use a particular uplink.  This is the only method presently supported.  In future releases you might be able to aggregate the bandwidth of multiple UCS vNICs to achieve a Port Channel style logical link.  This would work with IP Hashing on the vSwitch.

I understand that we can do NIC Teaming. Say in this case, I am using Active-Active NIC Teaming and Active-Passive NIC Teaming.

[Robert] You can use either A/A or A/P teaming on the vSwitch.  Most customers choose to utilize A/A to better balance bandwidth & traffic.

q2. It looks like there is ALWAYS 1-to-1 correspondence between what I  have created in UCS Service Profile and what vSphere had discovered, am I right?

[Robert] Correct.  In UCS you're defining virtual Hardware NICs for the ESX host.  Just like you can create multiple virtual NICs for a Virtual Machine where the VM's Guest OS will see these are distinct  interface cards, UCS does the same for the underlying Hypervisor Host - presenting up to 56 unique virtual NICs.

Thanks, again.

SiM

Regards,

Robert

Actions

Login or Register to take actions

This Discussion

Posted May 31, 2011 at 5:11 PM
Stats:
Replies:9 Avg. Rating:5
Views:954 Votes:0
Shares:0
Tags: No tags.
Categories: General UCS Hardware
+

Discussions Leaderboard