Connectivity failure when link from the bundle fails in a L2 etherchannel

themizzz21 · ‎01-25-2016

Hello,

On a C4506 switch [Catalyst 4000 L3 Switch Software (cat4000-I5K91S-M), Version 12.2(25)EWA14] we configured a Layer 2 etherchannel with 4 Gigabit ports with a vmware virtual switch.

=> Vmware suggests the following configuration on the Cisco switch( mode on ):

interface Port-channel1 switchport switchport access vlan 100 switchport mode access no ip address ! interface GigabitEthernet1/1 switchport switchport access vlan 100 switchport mode access no ip address channel-group 1 mode on

=> Bellow is the etherchannel as shown on the C4506 switch

B_R0a_Sw2#sh etherchannel 1 detail
Group state = L2
Ports: 4   Maxports = 8
Port-channels: 1 Max Port-channels = 1
Protocol:    -
                Ports in the group:
                -------------------
Port: Gi5/31
------------

Port state    = Up Mstr In-Bndl
Channel group = 1           Mode = On/FEC          Gcchange = -
Port-channel = Po1         GC   =   -             Pseudo port-channel = Po1
Port index    = 0           Load = 0x00            Protocol =    -

Age of the port in the current state: 00d:00h:01m:31s

Port: Gi5/36
------------

Port state    = Up Mstr In-Bndl
Channel group = 1           Mode = On/FEC          Gcchange = -
Port-channel = Po1         GC   =   -             Pseudo port-channel = Po1
Port index    = 1           Load = 0x00            Protocol =    -

Age of the port in the current state: 00d:00h:01m:01s

Port: Gi5/41
------------

Port state    = Up Mstr In-Bndl
Channel group = 1           Mode = On/FEC          Gcchange = -
Port-channel = Po1         GC   =   -             Pseudo port-channel = Po1
Port index    = 2           Load = 0x00            Protocol =    -

Age of the port in the current state: 00d:00h:00m:53s

Port: Gi5/45
------------

Port state    = Up Mstr In-Bndl
Channel group = 1           Mode = On/FEC          Gcchange = -
Port-channel = Po1         GC   =   -             Pseudo port-channel = Po1
Port index    = 3           Load = 0x00            Protocol =    -

Age of the port in the current state: 00d:00h:00m:24s

                Port-channels in the group:
                ---------------------------

Port-channel: Po1
------------

Age of the Port-channel   = 05d:23h:32m:47s
Logical slot/port   = 11/1          Number of ports = 4
GC                  = 0x00000000
Port state          = Port-channel Ag-Inuse
Protocol            =    -

Ports in the Port-channel:

Index   Load   Port     EC state        No of bits
------+------+------+------------------+-----------
0     00     Gi5/31   On/FEC             0
1     00     Gi5/36   On/FEC             0
2     00     Gi5/41   On/FEC             0
3     00     Gi5/45   On/FEC             0

Time since last port bundled:    00d:00h:00m:28s    Gi5/45
Time since last port Un-bundled: 00d:00h:02m:11s    Gi5/41

=====================================================

B_R0a_Sw2#sh run int gi5/31
Building configuration...

Current configuration : 145 bytes
!
interface GigabitEthernet5/31
description *** NEW BLADES ***
switchport access vlan 10
switchport mode access
channel-group 1 mode on
end

B_R0a_Sw2#sh int port-channel 1
Port-channel1 is up, line protocol is up (connected)
Hardware is EtherChannel, address is 0013.19f5.20fe (bia 0013.19f5.20fe)
MTU 1500 bytes, BW 4000000 Kbit, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, media type is N/A
input flow-control is off, output flow-control is unsupported
Members in this channel: Gi5/31 Gi5/36 Gi5/41 Gi5/45
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output never, output hang never
Last clearing of "show interface" counters 5d22h
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 1000 bits/sec, 1 packets/sec
5 minute output rate 39000 bits/sec, 5 packets/sec
     170895104 packets input, 13398135921 bytes, 0 no buffer
     Received 1131 broadcasts (23 multicast)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 input packets with dribble condition detected
     688972569 packets output, 976747229676 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier
     0 output buffer failures, 0 output buffers swapped out
B_R0a_Sw2#
B_R0a_Sw2#sh ru int port-channel 1
Building configuration...

Current configuration : 94 bytes
!
interface Port-channel1
switchport
switchport access vlan 10
switchport mode access
end

B_R0a_Sw2#

===========================================================================

Connected to the vmware virtual switch is a server.

The whole setup works OK.

The problem is that when an interface from the bundle fails(eg shut interface Gi5/31(or Gi5/36 or Gi5/41 or Gi5/45)), we have NO connectivity with the server.

When no shut interface Gi5/31 connectivity with the server is back again.

Thank you very much.

Mark Malone · ‎01-25-2016

have you configured the server correctly for nic teaming , also better to use lacp than mode on where possible , you can cause stp issues sometimes when forcing on just some options to check

themizzz21 · ‎01-25-2016

Hello Mark, thank you for the reply,

The nic teaming was done by other person. I think it's OK.

Vmware suggests the mode on.

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1004048

Is it due to the "mode on" that when a link from the bundle fails we lose connectivity with the server?

Thanks again.

Mark Malone · ‎01-25-2016

Hi no it shouldn't be but you should always use a protocol in etherchannel where possible rather than forcing on even if there example shows on ,protocols negotiate the connection rather than implying what it should be, I would say its the server side causing it ,theres nothing irregular with your config so unless your hitting a bug in the IOS its most likely the vmware setup

If you change it to lacp as a test and you get same issues with that as well as mode on then I would start checking the server side

themizzz21 · ‎01-25-2016

Hi,

Copying from the vmware knowledge base:

EtherChannel supported scenarios:

One IP to many IP connections. (Host A making two connection sessions to Host B and C)
Many IP to many IP connections. (Host A and B multiple connection sessions to Host C, D, etc)

Note: One IP to one IP connections over multiple NICs is not supported. (Host A one connection session to Host B uses only one NIC).
Compatible with all ESXi/ESX VLAN configuration modes: VST, EST, and VGT. For more information on these modes, see VLAN Configuration on Virtual Switch, Physical Switch, and Virtual Machines (1003806).
Supported Cisco configuration: EtherChannel Mode ON – ( Enable EtherChannel only)
Supported HP configuration: Trunk Mode
Supported switch Aggregation algorithm: IP-SRC-DST (short for IP-Source-Destination)
Supported Virtual Switch NIC Teaming mode: IP HASH. However, see this note:

===============

I see that only "mode on" is supported.

I will talk with the vmware technician and if there is nothing more to be done(from his side) i will open a TAC.

Thank you very much Mark.

amikat · ‎01-27-2016

Hi,

Will you please check that your vSwitch teaming settings (at vmware side) are as follows:

Load Balancing: Route based on IP hash,

Network Failover Detection: Link Status Only,

and no Standby Adapters configured.

Best regards,

Antonin

themizzz21 · ‎01-27-2016

Hello Antonin,

I will forward the settings you provided to the vmware tech in case something is missing.

Thank you very much,

Themis.

mark.chennell · ‎01-27-2016

Yes i agree with what has already been said, we always use LACP with your VM environment for reasons already stated, 9 times out of 10 when we see these issues it is always on the server side (its not our side, as we are network engineers ;0)