Cisco Support Community
cancel
Showing results for 
Search instead for 
Did you mean: 

"Etherchannel Loadbalancing on Catalyst Switches"

 

Introduction:

In this document you will learn about Ether-channel load balancing on Cat 6K, 7600, 4500, 3750.

Catalyst 6k and 7600:

 

How it is implemented on this platform?

 

The way EtherChannel load balancing works is that the switch assigns a hash result from 0-7 based on the configured hash  method ( load balancing algorithm ) for the type of traffic. This hash result is commonly called as Result Bundle Hash (RBH).

 

As for an example, let us consider that the port-channel algorithm configured on the box is src-dst ip then the source and destination IP of the packet will be sending to the hash algorithm (a complicated 17th degree polynomial) to calculate the RBH. Now each RBH is mapped to one unique physical port in the port-channel, whereas one physical port can be mapped to multiple RBHs (please look at following example for further clarification).

 

Let us consider the configured LB algorithm is src-mac and the switch is trying to send packets from 3 different src macs a, b & c over the ether channel( 5/1-2).

 

Now for packets from "a" the hash algorithm computes a RBH of 6, 5 for "b" and 4 for "c".

 

It is possible that RBH of 5/1 is mapped to RBH of 6 & 4, 5 for 5/2 but one RBH can be mapped to one physical port only. It is not possible that a RBH ( say 3) is mapped to both 5/1 and 5/2.

 

Things to check/how to check

 

------------------------------

 

1. What is the configured load balancing algorithm?

 

From the SP "show etherchannel load-balance"

 

e.g.

 

6500-20-sp#show etherchannel load-balance

 

EtherChannel Load-Balancing Configuration:

 

        src-dst-ip <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

 

        mpls label-ip

 

EtherChannel Load-Balancing Addresses Used Per-Protocol:

 

Non-IP: Source XOR Destination MAC address

 

  IPv4: Source XOR Destination IP address

 

  IPv6: Source XOR Destination IP address

 

  MPLS: Label or IP

 

6500-20-sp#

 

2. What is the RBH value chosen for the packet from 1.1.1.1 to 2.2.2.2?

 

From the SP " test etherchannel load-balance interface port-channel <port-channel #><mac/ip> <source address> <destination address>"

 

Choose the attributes of the above command based on the output of step 1. If the configured load balancing algorithm is src_ip then just give the src-IP of the packet 1.1.1.1. Since in our example the configured load-balancing algorithm is src-dst IP we will feed both 1.1.1.1 and 2.2.2.2 in the above command.

 

e.g.

 

 

6500-20-sp#test etherchannel load-balance int port-channel 1 ip 1.1.1.1 2.2.2.2

 

Computed RBH: 0x5

 

3. How to check which physical port is mapped to what RBH value?

 

In certain version of IOS the output of test etherchannel command will give you the physical interface chosen; this step (3) is when you do not get the egress interface information from the test etherchannel command output.

 

For the IOS where the command gives the egress interface the o/p will look like this.

 

6500-20-sp#test etherchannel load-balance int port-channel 1 ip 1.1.1.1 2.2.2.2

 

Computed RBH: 0x5

 

Would select Gi3/2 of Po1

 

For other versions go to the RP and type the command "show interface port-channel <num>  etherchannel". Look at the "Load" column o/p corresponding to a physical interface.

 

Convert the "Load" value into binary (look at the below example)

 

6500-20#show interface port-channel 1 etherchannel

 

Port-channel1   (Primary aggregator)

 

Age of the Port-channel   = 0d:01h:05m:54s

 

Logical slot/port   = 14/1          Number of ports = 2

 

HotStandBy port = null

 

Port state          = Port-channel Ag-Inuse

 

Protocol            =   LACP

 

Fast-switchover     = disabled

 

Ports in the Port-channel:

 

Index   Load   Port     EC state        No of bits

 

------+------+------+------------------+-----------

 

  0     55     Gi3/1    Active    4

 

  1     AA     Gi3/2    Active    4

 

Here the load value for gi3/1 is "55" and for gi3/2 is "AA".                  

 

                  7654  3210

gig3/2 - AA -1010 1010

                      |      | 

                    A       A

 

gi3/1 -  55  - 0101 0101

                   ------   -----                 

                     |       |  

                     5      5

 

for gig3/2 bits 1,3,5 and 7 are set. So RBH value of 1,3,5,and 7 will choose gi3/1.

 

for gi3/1  bits 0,2,4 and 6 are set. So RBH value of 0,2,4,and 6 will choose gi3/2

 

From the outputs you can observe that 4bits are set for each of the two interfaces. Hence when there are 2 links in the ether channel then each link has equal probability of getting utilized.

 

However for 3 links in etherchannel the test etherchannel output will look similar to this.

 

6500-20#show interface port-channel 1 etherchannel

 

Port-channel1   (Primary aggregator)

 

Age of the Port-channel   = 0d:01h:05m:54s

 

Logical slot/port   = 14/1          Number of ports = 2

 

HotStandBy port = null

 

Port state          = Port-channel Ag-Inuse

 

Protocol            =   LACP

 

Fast-switchover     = disabled

 

Ports in the Port-channel:

 

 

 

Index   Load   Port     EC state        No of bits

 

------+------+------+------------------+-----------

 

  0     49     Gi3/1    Active    3

 

  1     92     Gi3/2    Active    3

 

  2     24     Gi3/3    Active    2

 

Here the bit sharing ratio is 3:3:2 hence two links has higher probability of getting utilized as compared to the third link (more in the additional section at the end).

 

This table lists the ratios of the values that each port accepts, which depends on the number of ports in the EtherChannel:

 

8

1:1:1:1:1:1:1:1

7

2:1:1:1:1:1:1

6

2:2:1:1:1:1

5

2:2:2:1:1

4

2:2:2:2

3

3:3:2

2

4:4

 

Note: This table only lists the number of values, which the hash algorithm calculates, that a particular port accepts. You cannot control the port that a particular flow uses. You can only influence the load balance with a frame distribution method that results in the greatest variety.

 

We also support “per module load balancing” for DFC LC, where you can define LB algorithm per module basis.

 

http://www.cisco.com/en/US/docs/switches/lan/catalyst6500/ios/12.2SX/configuration/guide/channel.html#wp1020804

 

For this implementation we would need to keep in mind that the hash decision is taken on the INGRESS line card. If you have configured ether channel LB for the DFC where the actual physical link in the ether channel exist then your Load balancing might not work as you have desired as ingress LC will decide the egress physical port. By default any LC ( with or without DFC ) will load balance traffic based on the algorithm configured on the PFC. To check the “test etherchannel” command session to the DFC module and then issue the command.

 

For Cat 4500:

 

How it is implemented on this platform?

 

On this platform we use the concept of Agg2PhyPort mapping table. Agg2PhyPort table as the array of 8 elements, each can contain a port number, say for 2 ports a and b then {a,b}: .

 

Hash function will calculate the index into that array based on the input information: so it will be either 0 or 1 (index is 0 based).

 

Here’s an example:

 

Imagine you use 3 links in a bundle (say port 5, 10 and 20), then agg2phyport table would look like:

 

0: 5

 

1: 10

 

2: 20

 

max-length=3 (number of ports in a bundle)

 

Now, hash algorithm produces, say 7 (for configured input parameters), then index will be calculated as 7%3=1 and port 10 will be selected.

 

What to check/how to check?

 

 

1. How to check Agg2PhyPort mapping table?

 

"show platform mapping port" is the command however it is not worth doing it as the output of the command provided in step 4 gives you the egress port all the time.

 

2. How to check the o/p of hashing algorithm?

 

Not worth checking because of the above reason. The hash value for the 4500 is calculated via a 'rolling XOR' which is Cisco Confidential.

 

3. Check the configured load balancing algorithm by using the command "show etherchannel load-balance".

 

4. Use the command "show platform software etherchannel port-channel 1 map " to find the egress interface.

 

BGL-4500-12#show platform software etherchannel port-channel 1 map ip 1.1.1.1 2.2.2.2

 

Map port for Ip 1.1.1.1, 2.2.2.2 is Gi2/1(Po1)

 

NOTE: Software forwarded traffic will use Gi2/1(Po1)

 

While using the above command please keep in mind CSCtf75400(registered customer only)

 

If you hit this bug then unfortunately you have to rely on the sniffer capture to get the actual egress interface,

 

In K5 based architecture we have actually got rid of unequal load balancing problem when the number of links are 3,5,6 or 7 . As mentioned in the doc that we use 8 bits of hash result to  determine the load balancing, in a scenario where we have 3 physical links in the ether channel 3 bits will be chosen for link 1, 3 for link 2 and 2 for link 3. So the ether channel load balancing probability is ( 3:3:2). However in K5 we use only last 3 bits of the hash result ( for 3 links in EC , 5 bits for 5 links in EC , 6 bits for 6 links in EC and so on ) for 3 links in EC. This way we ensured that all the links in the EC has equal probability of taking the traffic. In K5, in order to improve load-balancing determination and flow distribution we stepped away from the “modulo” approach and load-balancing is based on the pre-programmed hardware mapping table.

 

For Cisco 3750:

 

 

On 3750 we use a similar 8 bit hashing algorithm and hence traffic distribution is more even when number of links in the ether channel is 2 4 or 8 ( please look at the common scenario section for details).

 

Command to check the egress interface in the port-channel is

 

" test etherchannel load-balance interface port-channel <port-channel #><mac/ip> <source address> <destination address>"

 

Common Scenarios:

 

-----------------------------

 

Ether channel not load-balancing properly?

 

To understand the scenario it is important for us to determine all the flows which the etherchannel is handling. Number of flows will depend on the configured load balancing algorithm. Let us take an example.

 

Source 10.0.0.1 (mac a.a.a ) sending a tcp stream to 15.0.0.1 ( mac b.b.b ) with a source tcp port of 50 and destination 2000

 

Source 10.0.0.1 (mac a.a.a ) sending a tcp stream to 15.0.0.2 ( mac c.c.c ) with a source tcp port of 60 and destination 2000.

 

If configured load balancing algorithm is SRC_MAC

 

Then no of flows = 1

 

If configured load balancing algorithm is DST_MAC

 

Then no of flows = 2

 

If configured load balancing algorithm is DST_PORT

 

Then no of flows= 1

 

and so on.

 

The ways you can capture the flows are:

 

-> Sniffer - difficult and hectic.

 

-> Netflow - relatively easier.

 

-> External monitoring tool.

 

Once you have a good idea of the flows then check which flow will take which physical interface. Use the tools discussed above to determine the physical interface.

 

This step will help you to explain why we see unequal distribution of traffic over the physical interfaces.

 

Here are the few scenarios which can cause unequal distribution?

 

1. Let us consider we have two flows and two physical interfaces in the etherchannel. It might be possible that one flow is more talkative than the other.

 

Now consider i have 5 flows out of which one is super talkative, this flow can overwhelm others. Whichever physical interface this flow is choosing will have relatively higher utilization than the others.

 

Resolution- flow control the super talker, need to look at from the host side.

 

2.  One very common problem is that you do not have enough number of flows and out of whatever small number of flows which you have most of them are hashing to the same physical interface.

 

Resolution- Increase the number of flows. Try changing the hashing algorithm to the one most appropriate to the traffic.

 

3. When you have 3, 5, 6 or 7 physical links in the ether channel then few links will have higher probability of taking the traffic than the other ( based on number of hashing bit assigned to each physical interface )and hence there is an inherent chance the  traffic will be unequally distributed.

 

Resolution - use 2, 4 or 8 numbers of links in the ether channel.

 

Related Information:

Understanding EtherChannel Load Balancing and Redundancy on Catalyst Switches

Version history
Revision #:
2 of 2
Last update:
‎08-23-2017 11:07 PM
Updated by:
 
Labels (1)
Contributors
Comments
New Member

very nice artile only part what i miss is as u said convert load value so as above load values are 55 annd AA binary value for 55 =00110111 and u tell  55  - 0101 0101 how and second this how u convert AA into binary pls explain so i get this concept.......................?

New Member

@Souvik Ghosh buddy  i thought u mistype G3/1 has value set 0,2,4,6 so RBH use G3/1 for those value not G3/2 same for G3/2 value set is 1,3,5,7 so it choose G3/2 for that value but u said it use G3/1.how pls explain

Use 55 to be converted as 5 and 5 which is 0101 and 0101.

DONT take it as 55(fifty five) and do the conversion.