How to remove switches from a fabric.

Unanswered Question
Jun 15th, 2010

We have a couple of MDS9120's and a SUP1 9509 in our fabric that we want to retire.   I'm pretty sure that I don't want to just shut down these switches, unplug the cables and call it good.   Also the 9509 is the principal for two of our larger VSAN's and one of the 9120's is a principal for another that we are still using.  Also, this same used to have an IVR zoneset which I think is gone, at least I don't see it when I look at IVR.  Current all the switches in the fabric are running 3.31c code.  Since we are planning moving to the 4.x we want remove these since they will not be supported.

So, what is the procedure to make another switch the principal of a VSAN, and once a switch is no longer a principal what is the procedure for cleanly removing it from the fabric?     If someone has some insights or can point to me to documentation that covers this, it would be greatly appreciated.

Thanks,

Jeff Blomendahl

University of Kansas Medical Center

I have this problem too.
0 votes
  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 0 (0 ratings)
Loading.
dynamoxxx Thu, 06/17/2010 - 04:15

Jeff,

here is paragrapth from "Cisco MDS 9000 Family Fabric CLI configuration guide"

About Switch Priority


By default, the configured priority is 128. The valid range to set the priority is between 1 and 254.
Priority 1 has the highest priority. Value 255 is accepted from other switches, but cannot be locally
configured.
Any new switch can become the principal switch when it joins a stable fabric. During the principal
switch selection phase, the switch with the highest priority becomes the principal switch. If two switches
have the same configured priority, the switch with the lower WWN becomes the principal switch.
The priority configuration is applied to runtime when the fcdomain is restarted (see the “About Domain
Restart” section on page 18-3). This configuration is applicable to both disruptive and nondisruptive restarts.

i would change priority on one of the switches that is staying right before the upgrade so when upgrades are done( fcdomain gets restarted), you should have a new principal switch.

jblomendahl Tue, 06/22/2010 - 13:04

Thanks for the information on that.  So it sounds like basically I need make sure that the switch that I want to be the principal of the VSAN has the highest priority (lowest numerically).   Two of the three switches currently have their priority set to 1 and 2 depending upon the VSAN.   The rest of the switches are set to the default of 128.  My thought would be to set the priority of the three switches that are going to be removed to be lower than the default in all of the VSANs and set the priority of the switch that I want to be the principal for the VSANs higher than the default.   So, I would imagine that shutting down the "old" principal switch would trigger a fcdomain restart, or would I be better off doing a nondisruptive fcdomain restart and then shutting down the switch? 

Also, from some second hand information, I have heard that we should isolate (set to VSAN 4094) all of the ports except for the E-ports on the switch prior to it being removed.  My understanding is that it reduces the amount of time that fabric is segemented after the switch is removed.  Any thoughts on this?

dynamoxxx Wed, 06/23/2010 - 18:15

i've done non-disruptive domain restart without issues, don't know about the port and fabric reforming time.

jblomendahl Thu, 07/01/2010 - 13:26

In case anyone is still interested,  I opened a TAC case on this because I could see that the information was all out there, but I wanted to make sure that I didn't miss anything that would bite me.  It turnes out that it wasn't that big of a deal if you do it correctly.

To remove a switch that is the principal for a VSAN, as was mention earlier in this thread, you need to give it a lower priority and give the switch that you want to make the principal the highest priority.  In our case, the "old" switch had a priority of 1 for 3 of our VSANs, so I changed it to the default of 128 and on the switch that was to be the "new" principal, the priority was set to 1.  Assuming that you are in config mode on the switch, the commands from the CLI looked like this:

On the "old" switch:

fcdomain priority 128 vsan xx   (xx being the number of the VSAN, this sets the switch priority to the default of 128)

fcdomain restart xx     (again, xx being the number of the VSAN,  This will non-disruptively restart the vsan on the switch and change the priority.  Be aware that if you are using static domain ID's the restart may be disruptive on the switch, check the documention for more on this)

On the "new" switch:

fcdomain priority 1 vsan xx   (xx being the number of the VSAN, this sets the switch priority to the highest priority, 1)

fcdomain restart xx     (again, xx being the number of the VSAN,  This will non-disruptively restart the vsan on the switch and change the priority.  Be aware that if you are using static domain ID's the restart may be disruptive on the switch, check the documention for more on this)

Do this for every VSAN that the "old" switch is the principal and also on the "new" switch that is to be the principal.  From what I saw in the Domain Manager this will change the priority but won't necessarily change the principal assignment.  At this point the "old" switch is ready to be removed from the fabric.  Obviously, since this switch was the principal, you'll want to make sure that all of the zoneset information for those zones has been replicated to the other switches in the fabric, particularly to the switch that is to become the new principal.  Other than an ISL to the rest of the fabric and the connections to the management ports, there was nothing connected to the "old" switch. 

At this point you can shutdown the ISL to the "old" switch from the switch that is on the other end of the ISL.  At this point you will see the fabric segment (as viewed from Fabric Manager)  At this point, I unplugged the management cables from the "old" switch, since I didn't want it talking to the Fabric Manager anymore.  I purged the switch and after about 5 or 10 minutes the segmentation went away and the fabric returned to normal, less the "old" switch.    There was no disruption to any of the 240+ hosts that were on the affected VSANs and the switch with the priority of 1 became the principal just like it was supposed to.

I think that once the ISL has been brought down, you want to take the switch off of the network or shut it down so that it can't continue to talk to the FM.  If this isn't done, FM will see that there are VSANs on that switch that are disconnected from the rest of the fabric and will show that the fabric is segmented.  I went through this procedure successfully on a SUP1 MDS 9509 and a MDS 9120 that were running 3.3.1 code.  One thing that did make my heart stop for a second was when I went to look at the zonesets after the segmentation cleared up.  The zonesets were there, but the zones were gone!   Since my pager wasn't vibrating out of the holster, I reasoned that the zones were still there.   A quick look at the configs on the switches confirmed this.  What had happened, was that the switch that I removed was the seed switch for FM when it discovered the fabric.  To correct this I restarted FM and rediscovered the fabric using the "new" principal switch as the seed switch, and all of the zones were back were they were supposed to be. 

Just for your piece of mind (read CYA), I recommend opening a TAC case just to have them look over your configs and make any recommendations.  In our case, we had to remove IVR from one of our switches and the engineer pointed out a bug that can occur when removing IVR, and a work around for this bug.

Sorry for the long post, but I hope that this information is helpful to someone else that has to do this.

Jeff Blomendahl

Enterprise Systems Engineer

University of Kansas Medical Center

wosuser2009 Thu, 06/24/2010 - 04:59

Quote:

"Also, from some second hand information, I have heard that we should isolate (set to VSAN 4094) all of the ports except for the E-ports on the switch prior to it being removed.  My understanding is that it reduces the amount of time that fabric is segemented after the switch is removed.  Any thoughts on this?"

Does this mean that the segmentation is a temporary thing?  the segmentation will right itself after some time? I am going through this exact thing right now.  Everytime i break the ISL link VSAN 1 segments.

Actions

This Discussion

 

 

Trending Topics: Storage Networking