Solved: advanced 7600 architecture question - port-channel hash distribution - balancing?

Shawn Kleinart · ‎07-21-2010

A few years ago I learnt that a port-channel (apparently any EtherChannel mode/config - I am using Gig links w/ LACP), either at layer 2 or 3, must have either 2, 4 or 8 members to distribute evenly across each. This is separate from the "port-channel load-balance" method of how to process traffic (flows - based on IP's, L4 ports, etc). I may have been sent an internal Cisco document (and have changed jobs and am trying to re-locate such information)... but if I recall correctly, the issue was the internal hash used by Cisco to distribute traffic over the bundle. As I recall, it was something like a 3-bit ID or hash used, so that there were 8 values that it would 'spray' too -- I hope I'm using fairly valid terminology here (?). So a value of 0-7 would be assigned to each member...

So p = packet...

Port

1 = p0, p2, p4, p6.. p8, p10

2 = p1, p3, p5, p7.. p9, p11

Or

1 = p0, p4

2 = p1, p5

3 = p2, p6

4 = p3, p7

... both are equally hit, same number of packets, etc.

Whereas, with 3 ports:

1 = p0, p3, p6

2 = p1, p4, p7

3 = p2, p5

.. and repeat the cycle. So link 3 gets 1/3 less traffic. In short, the bundle throughput is not 3Gbps, but roughly 2.6Gbps theoretical max before ports 1 and 2 would start dropping packets.

QUESTION:

Is this still true/valid? (even with the latest 7600 IOS and RSP720, in 3C mode)?

REF:

http://www.cisco.com/en/US/docs/ios/cether/configuration/guide/ce_lnkbndl.html#wp1068144

Load Distribution in an EtherChannel

Prior to Cisco IOS Release 12.(33)SRC, only a fixed load distribution algorithm was supported

The hash distribution (adaptive or fixed) is new to me, but appears to be a separate function of reprogramming on a bundle change. I don't have a free 7600 at this time to lab up and test/play with such configs and commands.

I haven't quite figured this feature out yet...

http://www.cisco.com/en/US/customer/prod/collateral/iosswrel/ps8802/ps6970/ps1838/prod_bulletin0900aecd805ddce3.html#wp9000872

Gigabit Ethernet LAG on UNI with Advanced EVC Load Balancing

On EVC port channels, the load-balancing hash algorithm uses EVC ID. In some cases, this may result in undesirable traffic split on port-channel member links. This feature will allow manual assignment of EVC to a particular link in a link bundle. This allows service providers to explicitly load balance the traffic from various EVCs.

Benefits

What is an EVC port channel? And EVC ID?

This explains fixed and adaptive well, and the hash schemes (which I do NOT think is specifically relevant to how the traffic is actually distributed on the port-channel itself, but maybe how it could be).

http://www.cisco.com/en/US/prod/collateral/switches/ps5718/ps9336/white_paper_c11_429338.pdf

I didn't find specific documentation that said you wouldn't get full rate or uneven distribution with 3 or 5 links... or that your channel should or must run 2 or 4 or 8 members either...?

Please advise. Anyone running a port-channel with 3 or 5 links and seeing a good traffic spread??

gephelps · ‎07-25-2010

Shawn,

Overall I see three parts to the question.

1) What is an EVC?

A quick description of EVC can be found here:

http://tools.cisco.com/squish/30af8

"Flexible QinQ Mapping and Service Awareness on Cisco 7600 Series ES+ line cards is supported only through Ethernet Virtual Connection Services (EVCS) service instances.

EVCS uses the concepts of EVCs (Ethernet virtual circuits) and service instances. An EVC is an end-to-end representation of a single instance of a Layer 2 service being offered by a provider to a customer. It embodies the different parameters on which the service is being offered. A service instance is the instantiation of an EVC on a given port on a given router."

The use of EVC requires Cisco 7600 Series ES+ and ES+20 Line Cards. In the same fashion a Port Channel using EVCs requires all of the ports to be on ES+ cards.

2) Port channel load hashing

You are correct in this description. When traffic is sent over the Port Channel, identification information is fed into a hashing algorithim and the result will be a value between 0 and 7 or the 3 bit number you refer to. This number will determine which link the traffic is sent across. You can see what values are assigned to each port with the show interfaces port-channel [number] etherchannel command. Here is the partial output from a 2 port bundle:

Index   Load   Port     EC state        No of bits
------+------+------+------------------+-----------
0     55     Gi5/2    Active    4
1     AA     Gi6/2    Active    4

In the number of bits column we can easily see how many values are mapped to this port. The load column represents which bits are assigned to each port. The load is actually individual hex digits. 5 is 0101 in binary and A is 1010 in binary. If I take the output from above and convert the load into binary it is easier to see:

Index   Load   Port     EC state        No of bits
------+------+------+------------------+-----------
0     0101 0101     Gi5/2    Active    4
1     1010 1010     Gi6/2    Active    4

Now each port has 4 bits set. If the result of the hash is 0, 2, 4, or 6 we will use G5/2. Assuming the traffic hashes perfectly into 8 values you will only get an even distribution with 2, 4, or 8 ports since we are always distributing 8 values. Here is an example of a 7 link Port Channel:

Index   Load   Port     EC state        No of bits
------+------+------+------------------+-----------
6     81     Gi2/1    Active    2
4     02     Gi2/2    Active    1
1     04     Gi2/5    Active    1
2     08     Gi2/6    Active    1
0     10     Gi2/9    Active    1
3     20     Gi2/13   Active    1
5     40     Gi2/14   Active    1

If we translate the load values again:

Index   Load   Port     EC state        No of bits
------+------+------+------------------+-----------
6     1000 0001     Gi2/1    Active    2
4     0000 0010     Gi2/2    Active    1
1     0000 0100     Gi2/5    Active    1
2     0000 1000     Gi2/6    Active    1
0     0001 0000     Gi2/9    Active    1
3     0010 0000     Gi2/13   Active    1
5     0100 0000     Gi2/14   Active    1

You can see only port G2/1 has two sets set.

3) What is the Load Distribution in an EtherChannel choice?

This change modifies which hash values are assigned to each port. The old method was to assign the value sequentially to each port in the bundle. So each time a link was added or removed to the bundle the distribution would change and all the ports had to be reprogrammed. In order to accomplish this with duplicating packets all of the hash values would be wiped and reprogrammed on all of the ports. This resulted in traffic matching each hash value would be dropped until that value was programmed to a port.

Without going into all the details this change makes the distribution of hash values more intelligent so that only specific hash values have a traffic interruption. As an example lets say you had two members in a bundle. Each member port would have 4 bits assigned to it out of the 8. When a 3rd port is added the distribution has to change. Now the first and second ports will have 3 values each and the new port will have two values. With the old distribution method the values were assigned sequentially so all 8 values had to be changed on all of the ports. With this enhancement only two values will be changed so traffic resulting in the two values being programmed into the 3rd port will be disrupted.

View solution in original post

gephelps · ‎07-25-2010

Shawn,

Overall I see three parts to the question.

1) What is an EVC?

A quick description of EVC can be found here:

http://tools.cisco.com/squish/30af8

"Flexible QinQ Mapping and Service Awareness on Cisco 7600 Series ES+ line cards is supported only through Ethernet Virtual Connection Services (EVCS) service instances.

EVCS uses the concepts of EVCs (Ethernet virtual circuits) and service instances. An EVC is an end-to-end representation of a single instance of a Layer 2 service being offered by a provider to a customer. It embodies the different parameters on which the service is being offered. A service instance is the instantiation of an EVC on a given port on a given router."

The use of EVC requires Cisco 7600 Series ES+ and ES+20 Line Cards. In the same fashion a Port Channel using EVCs requires all of the ports to be on ES+ cards.

2) Port channel load hashing

You are correct in this description. When traffic is sent over the Port Channel, identification information is fed into a hashing algorithim and the result will be a value between 0 and 7 or the 3 bit number you refer to. This number will determine which link the traffic is sent across. You can see what values are assigned to each port with the show interfaces port-channel [number] etherchannel command. Here is the partial output from a 2 port bundle:

Index   Load   Port     EC state        No of bits
------+------+------+------------------+-----------
0     55     Gi5/2    Active    4
1     AA     Gi6/2    Active    4

In the number of bits column we can easily see how many values are mapped to this port. The load column represents which bits are assigned to each port. The load is actually individual hex digits. 5 is 0101 in binary and A is 1010 in binary. If I take the output from above and convert the load into binary it is easier to see:

Index   Load   Port     EC state        No of bits
------+------+------+------------------+-----------
0     0101 0101     Gi5/2    Active    4
1     1010 1010     Gi6/2    Active    4

Now each port has 4 bits set. If the result of the hash is 0, 2, 4, or 6 we will use G5/2. Assuming the traffic hashes perfectly into 8 values you will only get an even distribution with 2, 4, or 8 ports since we are always distributing 8 values. Here is an example of a 7 link Port Channel:

Index   Load   Port     EC state        No of bits
------+------+------+------------------+-----------
6     81     Gi2/1    Active    2
4     02     Gi2/2    Active    1
1     04     Gi2/5    Active    1
2     08     Gi2/6    Active    1
0     10     Gi2/9    Active    1
3     20     Gi2/13   Active    1
5     40     Gi2/14   Active    1

If we translate the load values again:

Index   Load   Port     EC state        No of bits
------+------+------+------------------+-----------
6     1000 0001     Gi2/1    Active    2
4     0000 0010     Gi2/2    Active    1
1     0000 0100     Gi2/5    Active    1
2     0000 1000     Gi2/6    Active    1
0     0001 0000     Gi2/9    Active    1
3     0010 0000     Gi2/13   Active    1
5     0100 0000     Gi2/14   Active    1

You can see only port G2/1 has two sets set.

3) What is the Load Distribution in an EtherChannel choice?

This change modifies which hash values are assigned to each port. The old method was to assign the value sequentially to each port in the bundle. So each time a link was added or removed to the bundle the distribution would change and all the ports had to be reprogrammed. In order to accomplish this with duplicating packets all of the hash values would be wiped and reprogrammed on all of the ports. This resulted in traffic matching each hash value would be dropped until that value was programmed to a port.

Without going into all the details this change makes the distribution of hash values more intelligent so that only specific hash values have a traffic interruption. As an example lets say you had two members in a bundle. Each member port would have 4 bits assigned to it out of the 8. When a 3rd port is added the distribution has to change. Now the first and second ports will have 3 values each and the new port will have two values. With the old distribution method the values were assigned sequentially so all 8 values had to be changed on all of the ports. With this enhancement only two values will be changed so traffic resulting in the two values being programmed into the 3rd port will be disrupted.

Shawn Kleinart · ‎07-29-2010

Thanks so much, appreciate the detailed response!