Can someone help with this?
VSANS are like VLANs in that they isolate ports in the MDS switch fabric so that only the ports in the VSAN can communicate with other ports in the same VSAN?
What else does the VSAN do?
Zones are logical connections in the VSAN from point A to point B?
I have noticed that the Zones look like they are specifically the connection from the server to the disk space and there are only two connections in each Zone?
Is this correct?
What else is the Zone doing?
Sorry for the broad questions.
The alias looks like the wwn which has been given a user friendly name for easier remembering?
Is there an easy way to undestand these components?
Yes, you are correct that VSANs partition control and data traffic. They can be used to consolidate many SAN switches to just a few, such as allowing FICON (mainframe) and open systems to share the same switch. VSANs can also be used to add non-Cisco switches to a SAN.
The MDS Configuration Guide explains each feature such as VSANs and device alias.
By your questions I am guessing that you are a network engineer tasked with running a SAN. You are not alone, this is becoming a common occurrence lately.
Let me see if I can answer some of your questions.
Lets start a lower level.
A host that is connecting to storage is called an initiator
A host that is storage, is called a target
Depending on what type of devices is it can have many port types, though for the most part this will be automatically chosen for you.
These map into different functions, but the key takeaway is that N ports are usually your hosts, TE ports are usually links between your SAN switches and F / FL ports are almost always some type of storage.
When a host plugs into your SAN, A couple things happen. First thing is PLOGI (port log in) which determins the port type and speed. Then it does a FLOGI (Fabric Log In) at this time it presents the Fabric (name for your Storage Network) with is PWWN (port world wide node name). Your fabric then creates what is Called a FCID (Fibre Channel Identifier) that has two items in HEX. First item is what is called your domain (which is like a subnet) and the second item is unique value that can be hard set or not.
VSANs are like VLANs in the fact that the provide isolated ports for hosts to come together in. They go one step past this, by providing other services, such as Fiber Channel Name Services, independant for each VSAN. FCNS is like, a dns server for your SAN, and is important for over all SAN operation.
Zones are like private VLANs. The provided a secure connection between two ports or world wide node names within a VSAN. Zones are generally from initiator to target (host to storage) and generally include only two items, though they can include more.
Zones are normally groped into what are called ZoneSets which allow you to active (apply) changes to large numbers of devices at once. This is a non distruptive, network wide change that is part of SAN administration.
You can also do neat things with Zones like filter SCSI messages and make discs read only.
Alias's are just that, they are human readable expressions of pwwn's (port world wide node names) These can be distributed Fabric wide the cisco fabric services (CFS) and really help stuff make sense in production.
All in all all these little pieces come together to provide a way for your server to mount its disk.
One note, as a network engineer you will be drawn to the CLI. Cisco actually made a great GUI called fabric manager that is free with the MDS switch. I recommend that you try it out, the docs can be a little light at times, but the interface is super.
If you find this helpful please rate it
Colin Thanks for the great information.
You are correct, I am newly resposible for the SAN switchgear and I am trying to understand the components a little better.
A follow up question:
Can you give me an example of the benefit of having the services independant of each VSAN?
Where would the service be used in one VSAN and not used in another?
A good example of using vsans would be to strictly segregate production vs development traffic.
Another use, is say you have an IBM mainframe, along with some Windows boxes both utilizing the same fabric. The IBM servers will use a protocol called FICON to manage the fabric and storage, while your windows boxes will just mount it.
You would separate your windows hosts from your AIX hosts using VSANS and only run FICON on the IBM vsan.
Another common setup is to use VSAN's and inter vsan routing (IVR) to utilize a shared backup pool
The last main reason is that you use VSANS almost like subnets, when doing san extension. you would provision 3 vsans, one for each site, then a transit vsan for your wan link. That way if there was a connectivity drop it wouldn't cause instability in either site.
As you said the domain id is like a subnet, is the purpose of the domain id and fcid to ensure that traffic reaches source and destination in the FC network, this is the reason they have to be unique per vsan?
We have a 9509 in the HQ site connected via fcip link to DR site 9216.
There is one VSAN connecting the storage arrays between the two sites.
Everything at each respective site is connecting locally to their storage array, so no traversing VSANS and no IVR is configured. Both MDS switches show up in Fabric Manager with all of the connections.
I can see all five VSANS (two in HQ connecting HQ servers to the HQ array, two in DR connecting DR servers to Dr array).
I also only see local VSANs in each switch
Would this be considered one fabric or two seperate fabrics?
Heavy emphasis on "like" a subnet. Domain ID's are unique per Switch/VSAN. They can be dynamically assigned, statically assigned, or cached.
On a primary level you can look at your domainID as allowing for routing between switches in the same VSAN. (please no one jump on me about l2 vs l3, its all kind of grey in fibre channel)
Fabric generally refers to the "network" of san switches themselves. Vsans would just be a "thread" in that fabric.
So, this would be considered one fabric as long as it was over the same gear.
If you have no IVR configured, double check that your FCIP trunk is not passing VSANs accross the wan link.
The risk is, that whenever that WAN link drops, that it will force a fabric reconfiguration on each edge VSAN. (RCF message I think)
On thing that I didn't touch on, is that generally in storage, we have A and B fabrics. They are both independent and do not mix. This segregation generally holds true all the way down to the host level.
In this case, since fabrics A and B are not connected, then yes you would have two fabrics. It would stay two fabrics even if you had a 100 vsans running across it.
Thanks again for the info,
Can you help me understand the th tuning parameters on FCIP link we have from HQ to DR?
Currently we have the profile set up as such on a DS3 to DR that is being shared with other data going across the link:
fcip profile 1
ip address 10.1.82.1
tcp max-bandwidth-mbps 1000 min-available-bandwidth-mbps 30 round-trip-time-ms 4
tcp send-buffer-size 5000
Usually there is not much utilization (about 10%) from the other data.
My unsderstanding is that I need to adjust the Max bandwidth size to be closer to the actual upper end of what I will have on the link for the fc traffic.
I understand with the setting shown that fcip link thinks it has a 1G port but actually only has maybe 40Mbps and could possibly try to push more that the link can handle, but I am wondering if there are flow control paramters that by tuning this could make the link more efficient.
Is this the case?
At the same time we have had a couple of isolated incidences where the fcip link was flapping with the MDS switch throwing "tcp retransmit failure" errors that even by adjusting the "max bandwidth available" did not correct.
The DS3 was only about 30% utilized during one of these incidences, could that have been related?
But I am thinking that rather than cause the link to flap, it would be more of efficciency of the link and allow more throughput if it were tuned correctly with the min and max bandwidth (adjusting the window size).
You will like this whitepaper, it is a little old (from SanOS 1.x) but it does a really good job of explaining guidelines for configuring your FCIP interfaces.
You are correct in noting that this FCIP profile is not configured appropriately for a DS3 interface. This can cause many headaches.
Reference the link above for sizing guidelines for the different tcp values. It also goes into something called Buffer to buffer credits (B2B Credits) which you will start seeing in Datacenter ethernet pretty soon. These are in a way like fecn's and becn's. They are used for flow control between hosts. The thing is, when you seperate hosts of long distance, you get what is called "droop" where your B2B credits are litterally still on the wire, so you need to spoof them to get full throughput.
This link might be handy for you, there are alot of little tweaks that you can do around tcp windowing, encouraging fast start, etc, and they are detailed below.
SanOS 3.0 FCIP configuration guide -
On the subject of link drops, if you link was dropping and your CPE interfaces wheren't, the first thing I would look at the availability metrics between both sites (on that link).
If you don't have it already, it might be a good think to throw up a active probe, IP SLA or something along that line. If you don't you will probably end up chasing your tail because some transit AS decided to do maintenance.
One other thing that you can do if your provider supports QOS is to tag your data as it leaves the FCIP interface.
qos control 24 data 26
control frames would be marked with a DSCP value of 24, and Data frames with 26.
Hope this helps.
Sorry for the basic questions but:
How do I obtain availability metrics?
Where can I get information on setting up an active probe?
What is a "transit AS"?
Sorry, looks like I devolved into geek speek.
To get availability metrics, you need to monitor. e.g. ping the far side.
There are plenty of tools out there, including tools built into cisco routers. Though my favorite is nagios.
a transit AS just refers to a hop between providers in the internet (its a bgp term).
I have noticed that when I have experienced the fcip link drop, the response time has been 50ms as opposed to 3ms normally.
Since I do not have nagios, what other options are there for an active probe?
Are you talking about checking response time?
Colin is correct -
I once stomped production servers in the SAN because I was fooling around with the development Oracle servers. My guard was down because they were not "production" - whoops... Having seperated them in at least two VSANs would have saved me from myself.
The FICON example is good too. On the MDS, one could have FICON and Open Systems sharing the same switch. Many times this gives classical Mainframers all sorts of fits. By implementing VSANs and explaining the seperation of services and essentially architecting a veil within the switch, they may be more aimiable to sharing the resource of a larger consolidated director.
Now that you know how cool VSANs are, I hae one word of caution -- just because I can have many VSANs, does not mean that I should. This is definately a place to spend some quality time in architectural conversations and white board sessions. Think about how you want to manage your environment down the road. Use the beauty of VSANs as an advantage - for security, seperation and consolidation on the hardware levels.