VMWare ESX NIC Teaming

Unanswered Question
Jun 18th, 2010

Hello,

We've recently migrated some of our ESX servers to a 3-port NIC team design.  Everything seems from our end to work fine, however, we have had complaints that suddenly the VMs on those hosts are running "slow."

On the ESX server side, we set up "Route on IP hash" and everything else as recommended by several sources.

On the Physical Switch (3750 Stack):

interface Port-channel7
description NIC Team
switchport trunk encapsulation dot1q
switchport trunk allowed vlan xx,yy,zz
switchport mode trunk

and on the individual interfaces we have:
description Po7 Member 1
switchport trunk encapsulation dot1q
switchport trunk allowed vlan xx,yy,zz
switchport mode trunk
channel-group 7 mode on

I did a little more reading today and saw that some people recommend using "switchport nonegotiate" and "spanning-tree portfast trunk," as well as setting the global load-balancing method to src-dst-ip.

I have a strong feeling that the user's complaints of slowness might be due to the fact that the global load-balancing method is set to src-dst-mac on the physical switch, and I would like to try it with src-dst-ip. The problem is that I don't know if changing this on a production switch with multiple etherchannels (to other cisco switches and ESX nic teams) would cause an outage.  I know that fooling with switchport nonegotiate will probably kill the link for a few seconds... is nonegotiate really that necessary? Will changing the load-balancing method kill my etherchannels and cause network downtime?

  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 0 (0 ratings)
Loading.
Ganesh Hariharan Fri, 06/18/2010 - 05:11

Hello,

We've recently migrated some of our ESX servers to a 3-port NIC team design. Everything seems from our end to work fine, however, we have had complaints that suddenly the VMs on those hosts are running "slow."

On the ESX server side, we set up "Route on IP hash" and everything else as recommended by several sources.

On the Physical Switch (3750 Stack):

interface Port-channel7
description NIC Team
switchport trunk encapsulation dot1q
switchport trunk allowed vlan xx,yy,zz
switchport mode trunk

and on the individual interfaces we have:
description Po7 Member 1
switchport trunk encapsulation dot1q
switchport trunk allowed vlan xx,yy,zz
switchport mode trunk
channel-group 7 mode on

I did a little more reading today and saw that some people recommend using "switchport nonegotiate" and "spanning-tree portfast trunk," as well as setting the global load-balancing method to src-dst-ip.

I have a strong feeling that the user's complaints of slowness might be due to the fact that the global load-balancing method is set to src-dst-mac on the physical switch, and I would like to try it with src-dst-ip. The problem is that I don't know if changing this on a production switch with multiple etherchannels (to other cisco switches and ESX nic teams) would cause an outage.  I know that fooling with switchport nonegotiate will probably kill the link for a few seconds... is nonegotiate really that necessary? Will changing the load-balancing method kill my etherchannels and cause network downtime?

Hi,

With source-MAC address forwarding, when packets are forwarded to an EtherChannel, the packets are distributed across the ports in the channel based on the source-MAC address of the incoming packet. Therefore, to provide load balancing, packets from different hosts use different ports in the channel, but packets from the same host use the same port in the channel.

With destination-MAC address forwarding, when packets are forwarded to an EtherChannel, the packets are distributed across the ports in the channel based on the destination host MAC address of the incoming packet. Therefore, packets to the same destination are forwarded over the same port, and packets to a different destination are sent on a different port in the channel.

So with every different source or destination mac it will use a diff link. If there is a session started on a particular link for a source to a destination it will keep on using the same link until the session termiantes or link goes down.

Hope to Help !!

Ganesh.H

Remember to rate the helpful post

Actions

This Discussion