Re: Intermittent disconnects with 1000V

kgarreau · ‎11-02-2009

Ok, we have two ESX 4 servers configured right now, dual CNAs connected to two separate upstream switches, 5020-A and 5020-B. We have the 1000V setup what we believe to be the proper way. All of the networking is connected to the 1000V - Service Console, VMotion, NFS, and all the VM VLANs. What we see are intermittent disconnects on our NFS datastores and service console connections 10-20 minutes after we add a second vmnic into the uplink. If we remove one of the vmnics and go back to a single connection, everything returns to normal.

I've included relevant parts from a show run and show spanning-tree from the 1000V... just looking for some insight on where to look next. Thanks.

Nexus1000V# show run

...

switchname Nexus1000V
vlan 1,60,160,235-236,610,669

...

port-profile system-uplink
capability uplink
vmware port-group
switchport mode trunk
switchport trunk allowed vlan all

channel-group auto mode on sub-group cdp
no shutdown
system vlan 60,235-236,610,669
state enabled
port-profile ServiceConsole
vmware port-group
switchport mode access
switchport access vlan 60
no shutdown
system vlan 60
state enabled

port-profile NXNFS
vmware port-group
switchport mode access
switchport access vlan 669
no shutdown
system vlan 669
state enabled

...

Nexus1000V# show spanning-tree

VLAN0001

Spanning tree enabled protocol rstp

Root ID Priority 32769

Address 0200.0c00.003c

This bridge is the root

Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 32769 (priority 32768 sys-id-ext 1)

Address 0200.0c00.003c

Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Interface Role Sts Cost Prio.Nbr Type

---------------- ---- --- --------- -------- --------------------------------

Po1 Desg FWD 2 128.4096 Edge P2p

Po2 Desg FWD 2 128.4097 Edge P2p

VLAN0060

Spanning tree enabled protocol rstp

Root ID Priority 32828

Address 0200.0c00.003c

This bridge is the root

Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 32828 (priority 32768 sys-id-ext 60)

Address 0200.0c00.003c

Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Interface Role Sts Cost Prio.Nbr Type

---------------- ---- --- --------- -------- --------------------------------

Po1 Desg FWD 2 128.4096 Edge P2p

Po2 Desg FWD 2 128.4097 Edge P2p

lveth3/1 Desg FWD 4 128.1 Edge P2p

lveth3/4 Desg FWD 4 128.4 Edge P2p

lveth4/1 Desg FWD 4 128.1 Edge P2p

lveth4/5 Desg FWD 4 128.5 Edge P2p

VLAN0160

Spanning tree enabled protocol rstp

Root ID Priority 32928

Address 0200.0c00.003c

This bridge is the root

Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 32928 (priority 32768 sys-id-ext 160)

Address 0200.0c00.003c

Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Interface Role Sts Cost Prio.Nbr Type

---------------- ---- --- --------- -------- --------------------------------

Po1 Desg FWD 2 128.4096 Edge P2p

Po2 Desg FWD 2 128.4097 Edge P2p

lveth3/5 Desg FWD 4 128.5 Edge P2p

lveth3/6 Desg FWD 4 128.6 Edge P2p

VLAN0235

Spanning tree enabled protocol rstp

Root ID Priority 33003

Address 0200.0c00.003c

This bridge is the root

Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 33003 (priority 32768 sys-id-ext 235)

Address 0200.0c00.003c

Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Interface Role Sts Cost Prio.Nbr Type

---------------- ---- --- --------- -------- --------------------------------

Po1 Desg FWD 2 128.4096 Edge P2p

Po2 Desg FWD 2 128.4097 Edge P2p

lveth4/6 Desg FWD 4 128.6 Edge P2p

VLAN0236

Spanning tree enabled protocol rstp

Root ID Priority 33004

Address 0200.0c00.003c

This bridge is the root

Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 33004 (priority 32768 sys-id-ext 236)

Address 0200.0c00.003c

Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Interface Role Sts Cost Prio.Nbr Type

---------------- ---- --- --------- -------- --------------------------------

Po1 Desg FWD 2 128.4096 Edge P2p

Po2 Desg FWD 2 128.4097 Edge P2p

lveth4/4 Desg FWD 4 128.4 Edge P2p

VLAN0610

Spanning tree enabled protocol rstp

Root ID Priority 33378

Address 0200.0c00.003c

This bridge is the root

Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 33378 (priority 32768 sys-id-ext 610)

Address 0200.0c00.003c

Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Interface Role Sts Cost Prio.Nbr Type

---------------- ---- --- --------- -------- --------------------------------

Po1 Desg FWD 2 128.4096 Edge P2p

Po2 Desg FWD 2 128.4097 Edge P2p

lveth3/2 Desg FWD 4 128.2 Edge P2p

lveth4/2 Desg FWD 4 128.2 Edge P2p

VLAN0669

Spanning tree enabled protocol rstp

Root ID Priority 33437

Address 0200.0c00.003c

This bridge is the root

Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 33437 (priority 32768 sys-id-ext 669)

Address 0200.0c00.003c

Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Interface Role Sts Cost Prio.Nbr Type

---------------- ---- --- --------- -------- --------------------------------

Po1 Desg FWD 2 128.4096 Edge P2p

Po2 Desg FWD 2 128.4097 Edge P2p

lveth3/3 Desg FWD 4 128.3 Edge P2p

lveth4/3 Desg FWD 4 128.3 Edge P2p

sudks · ‎11-04-2009

Hi,

Nexus 1K does not support Spanning Tree Protocol. The 2nd port which you added should be part of the port-channel.

The Spanning tree information below is that from N1K ? AFAIK we dont have this option.

The Port-channel if connected to the same Upstream switch then it should not have the CDP subgroup option. It can be either on or active or passive mode only.

Robert Burns · ‎11-04-2009

+ Post the Interface configuration for each 5020 port where the CNA is attached.

Have you tried manually assigning the sub-group IDs?

As a test, for "VEM 3" for example

With both interfaces uplinked to the 1000v do the following:

int eth3/1

sub-group-id 0

int eth3/2

subgroup-id 1

+ After you do this post the output of "show int brief".

Thanks,

Robert

kgarreau · ‎11-05-2009

Thanks guys. I will get back to you once I'm able to relay from the customer, I appreciate the help!

-Kenny

cts_jay · ‎12-03-2009

Have you resolved this? I'm having a very similar issue with vPC-HM and would love to see what your fix was.

jasonfmic · ‎02-05-2010

I experienced a similar problem which may be related. There is a current bug where multicast will publish MACs through a non-pinned subgroup. The current work around is to disable igmp snooping. Try disabling it globally. If your issue goes away, you may be hitting bug CSCte44240.

cts_jay · ‎02-08-2010

yep, i was hitting that bug too. unfortunately there is also a bug with disabling igmp snooping. We found that disabling it, whether globally or for specific vlans did not actually disable igmp snooping in all cases. cisco confirmed there is a bug. the workaround was basically to enable/disable it several times and force a system switchover (presuming you have a redundant VSM) until it starts working. I belive the bug has been fixed, but there has not been a patch/upgrade release date yet

jasonfmic · ‎02-09-2010

That doesn't sound fun. Ours seemed to disable without issue. Do you happen to be running 1000v in production? We are but only with low impact servers for now.

cts_jay · ‎02-09-2010

no, this was in our lab where our company's software gets tested before it moves to production. unreliable multicast is a deal breaker for our application so i've had to migrate off the 1000v for the time being until its fixed.