Re: change of principal switch

ivybeccukbc · ‎10-21-2010

Hello,

We have to fabrics with MDS9513 directors.

Yesterday we got a change of principal switch message and the VSANs notice on one of the fabrics.

Everything continued to work normally.

Is this a normal occurance? What could be the reason? Is it a serrious issue?

TKX,

Ivy

Kris Vandecruys · ‎10-21-2010

Hello Ivy,

A principal switch selection can be triggered by a Build Fabric or Reconfigure Fabric event. Since you didn't notice any interruptions it's probably not the latter one, because that's a disruptive event. However, something happened that caused this. While in itself not a problem, you might want to figure out what the cause exactly was.

The 'show fcdomain statistics' and 'show fcdomain internal event-history vsan 1' will give you some insight into what exactly happened.

hth,

Kris

ivybeccukbc · ‎10-21-2010

Hello Kris,

Nice to hear you again!

Command:
FDBEMF1C1# show fcdomain statistics

VSAN 1
VSAN Statistics
        Number of Principal Switch Selections: 2
        Number of times Local Switch was Principal: 0
        Number of non disruptive reconfigurations: 2
        Number of disruptive reconfigurations: 0

VSAN 100
VSAN Statistics
        Number of Principal Switch Selections: 0
        Number of times Local Switch was Principal: 0
        Number of non disruptive reconfigurations: 0
        Number of disruptive reconfigurations: 0

VSAN 110
VSAN Statistics
        Number of Principal Switch Selections: 2
        Number of times Local Switch was Principal: 0
        Number of non disruptive reconfigurations: 2
        Number of disruptive reconfigurations: 0

VSAN 120
VSAN Statistics
        Number of Principal Switch Selections: 2
        Number of times Local Switch was Principal: 2
        Number of non disruptive reconfigurations: 2
        Number of disruptive reconfigurations: 0

VSAN 199
VSAN Statistics
        Number of Principal Switch Selections: 2
        Number of times Local Switch was Principal: 2
        Number of non disruptive reconfigurations: 2
        Number of disruptive reconfigurations: 0

FDBEMF1C1# show fcdomain internal event-history vsan 1

        VSAN 1 FSM History - Current State DM_VSAN_S4_FWD_RDI
        -----------------------------------------------------
        0001. VSAN 1 117861 after Wed Oct 20 14:41:33 2010
        0001. VSAN 1 MsgType VSAN_TO_PORT
        0001. VSAN 1 Details DM_IFC1_SEND_EFP fc1/19
        0001. VSAN 1 State   DM_VSAN_S4_FWD_RDI
        0001. VSAN 1 Event   DM_VSF_E10_DOMAIN_ID_LIST_UPDATED
        -----------------------------------------------------
        0002. VSAN 1 117859 after Wed Oct 20 14:41:33 2010
        0002. VSAN 1 MsgType DM_MSG_MTS_NOT_PRINCIPAL_SW_INFO_UPDATE
        0002. VSAN 1 Details [not available]
        0002. VSAN 1 State   DM_VSAN_S4_FWD_RDI
        0002. VSAN 1 Event   DM_VSF_E36_MTS_NOTIFICATION
        -----------------------------------------------------
        0003. VSAN 1 115934 after Wed Oct 20 14:41:33 2010
        0003. VSAN 1 MsgType VSAN_TO_PORT
        0003. VSAN 1 Details DM_IFC1_SEND_EFP fc1/19
        0003. VSAN 1 State   DM_VSAN_S4_FWD_RDI
        0003. VSAN 1 Event   DM_VSF_E10_DOMAIN_ID_LIST_UPDATED
        -----------------------------------------------------
        0004. VSAN 1 115932 after Wed Oct 20 14:41:33 2010
        0004. VSAN 1 MsgType DM_MSG_MTS_NOT_PRINCIPAL_SW_INFO_UPDATE
        0004. VSAN 1 Details [not available]
        0004. VSAN 1 State   DM_VSAN_S4_FWD_RDI
        0004. VSAN 1 Event   DM_VSF_E36_MTS_NOTIFICATION
        -----------------------------------------------------
        0005. VSAN 1 50052 after Wed Oct 20 14:41:33 2010
        0005. VSAN 1 MsgType VSAN_TO_PORT
        0005. VSAN 1 Details DM_IFC1_SEND_EFP fc1/19
        0005. VSAN 1 State   DM_VSAN_S4_FWD_RDI
        0005. VSAN 1 Event   DM_VSF_E10_DOMAIN_ID_LIST_UPDATED
        -----------------------------------------------------
        0006. VSAN 1 50050 after Wed Oct 20 14:41:33 2010
        0006. VSAN 1 MsgType DM_MSG_MTS_NOT_PRINCIPAL_SW_INFO_UPDATE
        0006. VSAN 1 Details [not available]
        0006. VSAN 1 State   DM_VSAN_S4_FWD_RDI
        0006. VSAN 1 Event   DM_VSF_E36_MTS_NOTIFICATION
        -----------------------------------------------------
        0007. VSAN 1 49697 after Wed Oct 20 14:41:33 2010
        0007. VSAN 1 MsgType DM_MSG_MTS_NOT_PRIN_SW_DOM_IDS_VALID
        0007. VSAN 1 Details [not available]
        0007. VSAN 1 State   DM_VSAN_S4_FWD_RDI
        0007. VSAN 1 Event   DM_VSF_E36_MTS_NOTIFICATION
        -----------------------------------------------------
        0008. VSAN 1 44099 after Wed Oct 20 14:41:33 2010
        0008. VSAN 1 MsgType VSAN_TO_PORT
        0008. VSAN 1 Details DM_IFC4_DOM_ID_AVLBL [No Interface]
        0008. VSAN 1 State   DM_VSAN_S3_WAIT_FOR_DOMAIN
        0008. VSAN 1 Event   DM_VSF_E4_DOMAIN_ID_AVAILABLE
        -----------------------------------------------------
        0009. VSAN 1 42717 after Wed Oct 20 14:41:33 2010
        0009. VSAN 1 MsgType DM_MSG_MTS_NOT_FABRIC_NAME_VALID
        0009. VSAN 1 Details 20:01:00:0d:ec:87:9d:41
        0009. VSAN 1 State   DM_VSAN_S3_WAIT_FOR_DOMAIN
        0009. VSAN 1 Event   DM_VSF_E36_MTS_NOTIFICATION

I cannot judge the severity of the problem...

Kind regards,

Ivy

Kris Vandecruys · ‎10-21-2010

Ivy,

DM_VSF_E10_DOMAIN_ID_LIST_UPDATED means that a new domain was added to VSAN1. This could also mean that a trunk config has changed which involves VSAN1 or a complete link has been flapped. This is just the output for VSAN1 of course.

You now know when the event occured: Wed Oct 20 14:41:33 2010. This means you can correlate with the 'show accounting log' what happened at that time, but you'd have to do it on all switches involved.

As you can see in the statistics, this has been non-disruptive.

If you really want a full root-cause analysis it's best to open a case and include a full 'show tech-support detail' output.

ivybeccukbc · ‎10-21-2010

Hello Kris,

I ran the "show accounting log" on all 8 directors at Mechelen and there is no entry on Wed Oct 20 14:41:33 2010.

Do we have to leave it for now?

Kind regards,

Ivy

Kris Vandecruys · ‎10-21-2010

It depends what kind of time you are willing to invest in something that has had virtually no impact. The fact there is nothing in the accounting log means that most probable this isn't caused by human interaction.

Is the `show fcdomain internal event-history` complete? It appears truncated

If you require root cause, a full analysis can be done in a Service Request because I do not recommend attaching a full show tech-support in a public forum such as this.