Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for
Search instead for
Did you mean:
UCS Fibre Channel SAN Considerations
This document captures the key aspects of connecting UCS to a Fibre Channel SAN to increase the chances of your UCS working with a SAN first time.
The outline of this document should provide a robust framework for you to develop your own methodology for UCS and SAN design, but additional detail has been provided to provide enough explanation and examples should you be new to these concepts.
Here's what you'll find in this document. This outlines a checklist of considerations for using Cisco UCS with a Fibre Channel SAN.
UCS End Host Mode
UCS Expansion Slot
UCS Boot Policy
UCS VIFs and VFCs
UCS FC Failover
Be sure to bookmark / RSS this document because it's alive! It will be updated and spawn individual documents for particular SAN array and switch types. I envisage there being a document each configuration - one for UCS + MDS + EMC, another document for UCS + MDS + NetApp, etc.
Get involved! This is a collaborative effort and you are warmly invited to contribute either by feedback (positive or otherwise) or contribution.
Got feedback? Make a comment at the bottom, it's the quickest way to get in touch. Or use the forum People bit to speak with the author.
Considerations - key points to discuss and agree on - note this document seeks to provide a framework/checklist: not every point is covered in detail, but key points are explained (want a complete guide? Contact the author).
SAN Performance Design - this is a basic connectivity guide
iSCSI SAN - not supported at the time of writing.
Your architecture is a basic UCS system with two 6120XPs, one 5108 chassis containing two 2104XP FEX and eight half-slot B200 M1 blades.
Your Operating System (on each blade) is VMware vSphere 4
I've written this to appeal to a wide audience rather than the propeller head geeks who probably know all this anyway :-) It's informal and tries to communicate the key concepts rather than disappearing down in to the weeds. I hope I achieved my goal, but in case I haven't I'll keep improving this document based on the feedback I get.
There are three audiences:
System architect who brings together UCS and SAN and produces a design and bill of materials.
Compute Subject Matter Expert (SME) who will be building/running the UCS.
Storage SME who will be building/running the SAN + Array. Assuming this person also managed the FC switch fabric.
The audience is assumed to have working knowledge of UCS and SAN switching and arrays. If you don't, then you'll have to look elsewhere for that (perhaps we can provide guidance to get that knowledge? If you need help, contact the author via the Support Forums).
Steve Chambers, Solution Architect, Unified Computing Practice, Cisco.
I work for Cisco's Unified Computing Practice - if you would like me or one of my colleagues to help you with this topic, contact me directly or talk with your friendly local Cisco account manager. We can help you with the following services, and more, to get your there faster first time.
Watch out for following on documents that provide blueprints for actual configurations based on these considerations.
This is not an official Cisco document, use it at your own risk, your mileage may vary, prices go up and down, red sky at night shepherds delight
Why connect UCS to a SAN? The B-series blades can have disks installed in them, so why not use those? The driver to not install the OS on the local disks is to achieve stateless computing, where each blade is stateless: this means that any blade can be deployed with a new identity and purpose then redeployed with a new identity and purpose. It also allows for very quick recovery from blade hardware failure. Stateless computing can be achieved with UCS by using techniques such as UCS Service Profiles and SAN Boot.
Service Profiles are logical representations of a compute node (or blade). They define three groups of characteristics:
It is critical, before you do anything, to understand how your LUNs map to Storage Processors and how those Storage Processors connect into the SAN, and how your hosts (the UCS blades) connect through the SAn to those LUNs. This can be a chicken and egg situation when it comes to mapping LUNs to Hosts, because you might not have configured the Hosts yet... but you can't configure the Hosts to see the LUNs if you haven't configured the LUNs... so how to do it?
First of all, you need to understand how LUNs map to Storage Processors in your array. Do you have an Active/Active or Active/Passive array? If you have an EMC array then your DMX is an Active/Active and your Clariion is an Active/Passive. Why does this matter? In an Active/Passive array your vendor might direct you to map LUNs to specific primary storage processors. This means that the best way to then access that LUN is via the primary storage processor: you can probably still access the LUN via the secondary storage processor, but there is an access performance cost because the traffic has to traverse an internal bus: worse, if something persists on accessing a LUN via the secondary storage processor then you will cause something called a LUN Trespass and it might occur that your LUN is re-assigned to the other storage processor. But if other devices still want to access your LUN via the original primary, then you get more LUN Trespass and deteriorate to something called LUN Thrashing where your LUN ping-pongs between storage processors and severely degrades the performance of your array.
vSphere 4 is a lot smarter than previous versions and is pre-programmed to understand the array you connect (without you telling it!) and how to handle Active/Passive arrays. The take away here is to let the technology work it out for itself, but be aware of how things work. Trouble only usually happens when us dumb humans try to make a system behave in a non-default manner when we don't know any better. You have been warned!
Remember that you have two separate fabrics. Each blade has two HBAs connecting to separate fabrics. There are at least two separate FC switches, one for each fabric. When you get to the array, what happens next depends on the array model.
If the array is low tier and has just a couple of FC ports, then you are likely to connect Fabric A to Storage Processor A (or 0) and Fabric B to Storage Processor B (or 1).
If you have more ports, like four, then you are likely to connect each switch to a port on each SP - ie. for high availability, each fabric can see both SPs.
For high end devices like HDS99xx and EMC DMX-x, you have ports coming out of your ears and it's likely you don't need to worry about it :-)
Key takeway: understand the behaviour of your array, but let vSphere handle it automatically
With that out of the way, you need to work out how to limit access to your LUNs. There are two considerations here: boot LUNs and data LUNs.
Boot LUNs - you only ever want one host to access one Boot LUN at any one time. Unless you are particularly charitable in nature and expect everyone to play by the rules, then you should mask/map your LUN to one specific host. You can do this on the array when you know the port world-wide names of your host. But you don't know that until you've configured your UCS system so leave this for now.
Data LUNs - when running vSphere, these are the LUNs that will store for VMFS file system. VMFS is a concurrent access filesystem so, unlike a boot LUN, you want more than one host to have access. You are likely to configure a bunch of LUNs for a cluster, so you are likely to want to define a cluster of hosts and a cluster of Data LUNs (VMFS) and make sure each LUN is masked/mapped to the cluster of hosts.
FC Switch Considerations
Consider and explore the following checklist, more detail below.
What zoning will you use? Single-initiator zoning is best for VMware vSphere.
Will you be using VSANs? Only available on Cisco MDS and great at reducing under-utilized islands of FC fabric.
to UCS and Arrays, both logical (VSAN) and physical (port).
More detail on Switch Zoning
Zoning is crucial for correct functioning and adequate performance of your system. For VMware deployments you should use single initiator zoning: this is a really simple concept of configuring the SAN switch to only let your HBA see the storage processors it needs - and not other devices on the network, such as other hosts' HBAs.
Remember that you have two HBAs connected to two separate, "air gapped" fabrics which means each HBA connects to a different FC switch which means to each fabric your host is represented by only one HBA: hence, single-initiator zoning.
The key to successful zoning is not being lazy or crazy :-)
DO NOT configure a "cluster zone" with all HBAs for hosts in a cluster and the storage processors in one zone. That's lazy. All you need to get problems is for one host to start issuing BUS RESETS on the SAN and all your hosts will crawl to a halt.
DO NOT avoid zoning completely. That's crazy. Your host will be able to see every device on the SAN and be affected by what they do.
More detail on Swich VSANs
You only get VSANs on Cisco's MDS, not McData nor Brocade. VSANs are simple and similar to VLANs. Here's some considerations:
Currently each FC port northbound of UCS (ie. the port from the UCS fabric interconnect GEM to the FC switch) can only be in one VSAN. You can have one FC port in one VSAN.
The VSAN that your UCS connects into can be different than the VSAN that contains your FC Array if you use Inter VSAN Routing (IVR). Again, just Cisco MDS.
You tell UCS which VSAN your northbound FC ports are in, and you also connect HBAs to VSANs - all via the UCSM GUI, and more on that later.
More detail on Switch NPV
Because UCS is a really big host with more blades than FC ports, it means it has to "trunk" multiple-blade FC traffic down those over-subscribed ports while still uniquely identifying each blade (HBA) on the FC SAN. This is what NPV does, and all you really have to worry about is that your FC switches are NPV capable and that NPV is turned on.
Remember you have one switch per fabric UNLESS you are using Cisco MDS which means you can use VSANs.
If you aren't using VSANs, then the connectivity is quite simple - UCS northbound to a switch port, then switch ports to one or more storage processors on the array. Do your zoning based on pWWNs (HBAs and SPs).
If you are using VSANs, then the connectivity is still simple! A UCS port cannot (currently) "trunk" VSANs over one FC port, so you still only get one VSAN per port out of UCS.
Consider and explore the following checklist, more detail below.
the UCS HBA drivers are included in ESX, but is ESX qualified against your array? Check here.
UCS VIFs and VFCs
how the UCS switchboard works.
UCS FC Failover
HBA, FEX, FI etc
from Blade through the FEX and the 6120s to the switch to the array - putting it all together.
Quality of Service of FC traffic through UCS
More detail on UCS Identity
Each blade has a mezzanine card in it containing a Converged Network Adapter (CNA), and that CNA has Qlogic/Emulex/Cisco HBAs that are configurable in the BIOS and presented to the Operating System.
These HBAs need unique port World Wide Names (pWWNs) to identify them on the FC Fabric. In UCS, you could use the hard-burned in pWWN, but for stateless computing you instead use a logical Service Profile to tell the HBA what it's name is. You can do this manually per-Service Profile, or you can use a pool.
Use a pool to allow the Service Profile to pick a unique identity for itself, and remember this - it's what you're going to need to do your Array Mapping/Masking to expose LUNs to the HBA.
More detail on UCS vHBAs
We've already spoken (above) about identifying your UCS blade (HBA) on the SAN, but there's more to a HBA than just it's identity. Think about the following:
Do you need to tune Queues?
Do you need to tune Interrupt Handling?
Do you need to turn off Performance Enhancements?
Do you need to turn on Receive Side Scaling?
Do you need to tune Failover Timeout?
The answer to all of the above is No. But the last question is interesting - how does failover work? (See below!)
More detail on UCS Boot Policy
The blade BIOS dictates the boot order, and UCS Service Profiles configure the boot order via a Boot Policy, which you set in UCSM. Once you attach a Service Profile to a blade, then UCS configures the BIOS as per the Boot Policy.
It's normal (not sure if it's best?) practice to put the boot order like this:
In our case, you need to configure to boot from the HBA - but how does the HBA know where to boot from?
More on VIFs and VFCs
More on FC Failover
More on UCS Connectivity (putting it all together)