We have two sites about 1000 miles apart. Each site has virtual tape library connected to a Cisco MDS9222i. Each of the switches also has (2) GigE ports and the FCIP licenses. We created an FCIP tunnel and tried to use IVR, but the fabrics merged. I must have missed a step. I plan to break the ISL that was created through the FCIP tunnel, and just use IVR for FCIP without causing the fabrics to merge. Essentially, we just want the replication ports on each of the VTLs to see each other. Is there anything I should watch out for when severing the ISL? Also, are there any specific steps I need to follow to accomplish what I want to do?
I am attaching a simple visio diagram to help show I am looking to do.
Thanks so much in advance!
You can use IVR, that is not a problem and is recommended for what you are trying to do. Best practice would be to create a "transit VSAN" that the FCIP links use, for example it could be VSAN 20 or whatever. When you create your FCIP tunnels you want to be careful and make sure your not creating a trunk that is allowing all VSAN's. You will want to use the "switchport trunk allowed vsan" command to restrict the link to only allow the transit VSAN, for example VSAN 20.
Your going to have to do some reading on IVR no doubt, but what your trying to do is not very complex as the MDS has alot of the technology built right in to accomplish this sort of thing.
You can use VSAN 10 at both ends, but you end up with what is called "overlapping VSAN's", not a problem. you have to make sure when you build the IVR topology database that you use multiple AFID's. Recommended best practice would be as follows for example:
AFID1: Cisco 9222i SiteA VSAN 10
AFID2: Cisco 9222i SiteA VSAN 20 (or whatever the transit VSAN is you create)
AFID2: Cisco 9222i SiteB VSAN 20
AFID3: Cisco 9222i SiteB VSAN 10
I would use different domains on each switch in VSAN 10. You can use the same domain if you wish/have reason to, but then you really should enable ivr nat functionality.
So a few key things:
1. Create a transit VSAN
2. Build a proper IVR VSAN topology database using AFID's/VSAN's
3. Restrict your FCIP links to the transit VSAN
4. Create IVR Zones/zoneset and activate
MDS Configuration Guide should be able to point you in the right direction feel free to check back if you have issues.
Thank you so much for you're reply! I am glad that you say it is not as complex as I thought it was. I am relatively new to Cisco switches, most of my experience has been with Brocade.
I do believe I want to use a transit VSAN, it is my understanding, that if you don't use transit VSANs, you can potentially cause temporary interruptions across the fabrics. From what I have read, transit VSANs will ensure that only replication traffic (in our case), will travel through the FCIP tunnels. All other unecessary fibre channel traffc would remain local. I am still a bit confused about the transit VSAN... In my drawing it shows that the fabrics at both sites have VSAN10, this VSAN contains the Virtual Tape library. If I create a transit VSAN, let's say VSAN 200 which interfaces should belong to that VSAN? Just the FCIP interfaces and the VTL ports to be used for replication? For example, like this:
SITE A SITE B
The FCIP ports used for replication will reside in VSAN200 (at both sites) and also any VTL ports required for replication will be removed from VSAN10 at both sites and placed in VSA200.
Here is something else I should mention, which may or may not required changes. Currently at SITE A, there is a backup server in VSAN20 that accesses a tape library in VSAN10. IVR with NAT enabled is used to share the the tape drive with the backup server on VSAN20. There is essentially one IVR zoneset that has the HBA for the backup server in VSAN20 and the two FC ports for the tape library in VSAN10.
Will the existing IVR zonesets make any difference when reconfiguring the fabric to create the transit VSAN? Also, since the fabrics have already merge, anything I should look out for when removing the ISLs?
I really appreciate your help, it is definitely starting to make more sense now. I am just still confused as to which interfaces should belong in the transit VSAN.
Thanks for all your help Brian and everyone else!
Heather, you are correct that using a Transit VSAN will minimize the amount of disruption a WAN link can cause other VSANs, so the idea is the Transit VSAN is used to isolate the WAN.
Transit VSAN's in themselves do not determine what traffic can or cannot go over link. Its just a place to put objects. Normally the only members in a Transit VSAN are the WAN ports, which are typically FCIP ports (VE's) and are of course trunking.
Ok, so if you create a VSAN, put the FCIP ports in it, and connect the switches, you will potentially merge fabrics because by default trunks trunk all VSAN's, you must limit this, by doing "switchport trunk allowed vsan x", where vsan x is the transit VSAN.
If you do this, then you will NOT merge, because only the transit VSAN is allowed over that link, and the only things in that VSAN are the two (or more) FCIP ports.
So realize this. Traffic gets from one switch to another because its either a) trunked or b) routed via IVR. When you merged the first time you trunked the traffic, that was the mistake. We will enable the "switchport allowed" command and now we have essentially created a situation like:
VSAN10 <---->VSAN20 <---->VSAN30
Yes you can have VSAN10 at both ends of the link, but as we pointed out its best to use a totally new VSAN. Unless you have some reason you feel you need to make the VSAN's the same at your local and remote site, normally you would want to avoid this.
Bottom line, is regardless, you have members in separate VSAN's, these cannot talk, they are isolated as if they were in physically different SAN's. The only way to allow them to talk, is to use IVR. We specifically configure two things 1) which VSAN's are going to participate in the IVR topology and 2) more specifically which specific members of those VSAN's can talk to which other specific members of which other VSAN's in the topology.
Only your FCIP ports should be in the Transit VSAN200 you propose. The VTL's should stay in a non-transit VSAN. This can be VSAN10 if you like, but if you have two tape libraries one local and one remote, I would recommend putting the remote one in a new VSAN you create for the remote side that is not present on the local side.
So lets say you have something like this:
local site transit link remote site
VSAN10 VSAN200 VSAN20
you would create an IVR topology with a single AFID of 1 (common practice) and make sure all three VSAN's are in this topology. Then once you have this done you can create IVR Zone's and Zoneset's to decide who can talk to who. It sounds as if you already have an IVR Zoneset and IVR Zone, so you would just be creating some new IVR Zones and activating them.
What you describe where your tape is in a separate VSAN is quite common and that's fine. So at the local site you have VSAN10 and VSAN20 already in use, maybe you use VSAN30 for the remote site or some other number of your choosing. I would not use VSAN10 or VSAN20 if you can help it as it just makes things confusing. I would also use unique domain id's for all switches as well, statically assigned, I just like to have everything deterministic like that.
The existing IVR Zonesets do make a difference in the sense you want to be aware of what VSAN's already exist in the topology and just be familiar with what is currently going on in IVR just as you should with all active zones in the SAN environment.
Dont' feel you have to digest it all at once, I would go in baby steps, I would do the following:
1. decide on a VSAN to use for the remote site
2. decide on a VSAN to use for the Transit link
3. Create FCIP links, keep them shut down
4. add the switchport trunk allowed command to only allow the transit VSAN
5. you can assign the ports of the fcip to the transit vsan too, but not necessary as these FCIP links are ISL's (VE)
6. You can no shut the FCIP links they should come up, trunking the single VSAN which has no members other then themselves
7. Look at your IVR topology configuration, make sure it makes sense, if not you can post it here
8. Add the new transit and remote VSAN's to the topology and activate it, in more complex scenerios you may need/want to use IVR service groups but I am doubting that from what I am hearing so far, but you can always tell us all about your IVR topology.
9. Create new IVR Zones for the new members that need to talk, add these to the IVR Zoneset, Activate
As always, you want to make sure you have windows to do these things, if you have access to a lab, its always a good way to test, If there is any step your stuck on just post it here.
You also can use SAN Extension Tuner (if its supported on your hardware) to test the link. FCIP should be tuned properly for best performance, this can mean setting various tcp parameters in the fcip profile, and SET can help you accomplish that. You may also wish to use compression and other features such as write-accelleration or tape-accleration.
You can read a bit about SAN Extension Tuner on my blog at: http://www.feeny.org/?p=233
IVR is one of the more interesting features of the MDS's which Cisco was doing long before everyone else.
My response will focus on what you need to do to prevent your fabrics from merging. The easiest way to accomplish this
is to have different VSANs on each side (to avoid confusion) and have your FCIP tunnel be in a transit VSAN. I'll try and
show this via ascii art.
Site A Site B
(VSAN 10) (VSAN 20)
<----- FCIP ----->
You'd need to create VSAN 30 at each site, create your FCIP tunnel, then you port channel, make sure only VSAN 30
is allowed on the transit VSAN, and then create your IVR zones and zoneset.
There is another less complex way to do this but I'm not sure if you'll like the results but here is goes.
Site A Site B
(VSAN 10) (VSAN 20)
<----- FCIP ----->
Scenerio B wouldn't merge, as there would be members at each site only in one or the other VSAN. But the problem with not using a transit VSAN is of course the ability for WAN irregularities (very common) to cause stability issues with the fabric, a transit VSAN helps to isolate this.
Heather, if you don't HAVE to have the same VSAN at both sides, then Gary is correct in saying its best to use like 10,20,30, vs. say 10,20,10. I only used 10,20,10 in my example because your diagram showed it, and it is possible, but its always best to avoid overlapping if you can.
actually what I meant was that they will merge but it should be good merge as there should not be descrepencies (zone conflicts, etc). I would avoid the whole concept of using the VSAN on the WAN and local network though.
Heather, The transit VSAN is the best way to go as Brian pointed out. Our Primary and DR sites are only 20-22 miles apart and I'm using a transit VSAN for the very reason Brian pointed out.
Your correct. That is why I mentioned I didn't think Heather would like that scenario.
It does prevent the contents of Site B's VSAN 20 (formally VSAN 10) from merging together
with Site A's VSAN 10 and vice versa while keeping an AFID of 1 and NOT using IVR NAT
(I'm very anti-IVR NAT !!!!). My goal is to keep each fabric seperate while not introducing
to much complexity and IVR NAT. So Heather would end up with 1 fabric but each site
would be separated by VSANs.
I hope this makes sense.
Gary ..please excuse my lack of knoweldge in IVR functionality
So if i have VSAN 10 on one side and VSAN 20 on another site, without using transit VSAN (that would contain ports for both VTLs), how would i make two devices talk to each other ? Can i still use IVR without transit VSAN ?