Load balancing and Layer 2 preservation

[brian.k.clarke]

A customer currently had two data centers in different locations, connected via dual 50-Mb carrier circuits in a port channel between single switches on each end.  (Thus, there is no current routing between sites, and both sites create a single Layer 2 data center fabric.)

DC-A (172.16.10.x/24) -->  SwitchA  ======== dual port-channeled carrier links ======== SwitchB <---  DC-B (172.16.10.x/24)

The problem is that true load balancing isn't being achieved, so one of the circuits is getting crushed while the other lies idle. We need to implement a solution so that both links can be used closer to full capacity, and the customer can get closer to the full 100-Mbps of available combined circuit bandwidth.  Additionally, we need to preserve the ability for hosts on each end to reach the other using a common Layer 2 domain.  I need recommendations on how this could be accomplished, and I'll throw out some possible considerations:

1) somehow modify the port channel load-balancing algorithms on the current switches, so that some traffic prefers one link over the other.  One problem here is that the majority of the traffic congesting the first link is between two SAN systems (so, one source IP/MAC and one destination IP/MAC).

2) implement a single (or multiple?) routers on each end, and use routing protocols to handle the load balancing.  This breaks the L2 data center, though, unless some form of tunneling can be performed to unify the two sites.

The customer is open to purchasing new gear if that provides the optimal (or only) solution.

Thank you!

[deyadav]

As you have already figured out one particular flow which is being most utilized and is causing 1 link to be more utilized than the other, based on Source IP/MAC and Dest IP/MAC.

Well the etherchannel algorithm works on hash algorithm, which uses information in the packet/frame header to choose the link. So, if the hash is same the traffic would always traverse over the same link, we have no control over it.

We need to find out ways, we could use information in the packet header which changes within the same flow, to create different flow. For example, if L4 port number varies between the identified flow, then we change the load-balancing to use that information to have different flows.

http://www.cisco.com/en/US/partner/docs/switches/lan/catalyst6500/ios/12.2SX/configuration/guide/channel.html#wp1020804

On my device with SXI IOS version, I could see these options:

Switch(config)#port-channel load-balance ?
  dst-ip                 Dst IP Addr
  dst-mac                Dst Mac Addr
  dst-mixed-ip-port      Dst IP Addr and TCP/UDP Port
  dst-port               Dst TCP/UDP Port
  mpls                   Load Balancing for MPLS packets
  src-dst-ip             Src XOR Dst IP Addr
  src-dst-mac            Src XOR Dst Mac Addr
  src-dst-mixed-ip-port  Src XOR Dst IP Addr and TCP/UDP Port
  src-dst-port           Src XOR Dst TCP/UDP Port
  src-ip                 Src IP Addr
  src-mac                Src Mac Addr
  src-mixed-ip-port      Src IP Addr and TCP/UDP Port
  src-port               Src TCP/UDP Port

So, please consult the configuration guide for the switch in use, and find out which information varies often to utilize the above options to load-balance the traffic.

HTH.

Regards,

Deepak

[corey]

We're looking to do the exact same thing you're talking about.  We currently have 2 100 Mbps links in an port channel, but aren't able to effectively load balance with the methods available in the 3750's we're using on both ends.

Have you made any headway or found anything that will do what you need?

[jeffjadryev]

Hello,

Is one of the two links of the etherchannel 100% idle? That seems unusual to me...I would check to confirm that etherchannel is functioning properly if one of the links was 100% idle.

If you've already confirmed this, I think what Deepak suggested is the way to go. Etherchannel does a math function (XOR) on parts of the source and destination MAC addresses, by default, to decide which port in the channel to use to send a given frame. Given this, traffic between two hosts will always use the same port in the channel.

I'd use a sniffer/protocol analyzer at one of the two data centers to see what the traffic flow between hosts across the link is like. By looking at L2, L3 and L4 traffic summaries you might be able to deduce a better criteria for load balancing (again, per Deepak's post).

Hope this helps!

Jeff