Load balancing etherchannel

gsantini · ‎03-04-2008

Hi. I have a problem with an unequal load balancing between Catalyst 5500 with a Route Switch module and Cisco 7609.

I was configured an etherchanner on 5500 with 4 FE interfaces and these interface is a trunk with Cisco 7609. On the orher side, I was configured the same. 4 interfaces as a trunk.

Here is the partial configuradion about the interfaces:

Catalyst:

...

set port channel 8/17-20 desirable

set port speed 8/1-4,8/6-10,8/12-24 100

set port duplex 8/1-4,8/6-10,8/12-24 full

clear trunk 8/17 1001-1005

set trunk 8/17 on dot1q 1-1000

clear trunk 8/18 1001-1005

set trunk 8/18 on dot1q 1-1000

clear trunk 8/19 1001-1005

set trunk 8/19 on dot1q 1-1000

clear trunk 8/20 1001-1005

set trunk 8/20 on dot1q 1-1000

...

Cisco 7609

port-channel per-module load-balance

port-channel load-balance dst-mac

interface Port-channel4

switchport

switchport trunk encapsulation dot1q

switchport trunk allowed vlan 1,28,54,124,199,500-502,600,603,606,950

no ip address

load-interval 30

interface GigabitEthernet9/37

description lsevelez2 8/17

switchport

switchport trunk encapsulation dot1q

switchport trunk allowed vlan 1,28,54,124,199,500-502,600,603,606,950

no ip address

load-interval 30

speed 100

duplex full

wrr-queue cos-map 2 2 3

wrr-queue cos-map 3 1 4

wrr-queue cos-map 3 2 5

priority-queue cos-map 1 6 7

channel-group 4 mode desirable

!

interface GigabitEthernet9/38

description lsevelez2 8/18

switchport

switchport trunk encapsulation dot1q

switchport trunk allowed vlan 1,28,54,124,199,500-502,600,603,606,950

no ip address

load-interval 30

speed 100

duplex full

wrr-queue cos-map 2 2 3

wrr-queue cos-map 3 1 4

wrr-queue cos-map 3 2 5

priority-queue cos-map 1 6 7

channel-group 4 mode desirable

!

interface GigabitEthernet9/39

description lsevelez2 8/19

switchport

switchport trunk encapsulation dot1q

switchport trunk allowed vlan 1,28,54,124,199,500-502,600,603,606,950

no ip address

load-interval 30

speed 100

duplex full

wrr-queue cos-map 2 2 3

wrr-queue cos-map 3 1 4

wrr-queue cos-map 3 2 5

priority-queue cos-map 1 6 7

channel-group 4 mode desirable

!

interface GigabitEthernet9/40

description lsevelez2 8/20

switchport

switchport trunk encapsulation dot1q

switchport trunk allowed vlan 1,28,54,124,199,500-502,600,603,606,950

no ip address

load-interval 30

speed 100

duplex full

wrr-queue cos-map 2 2 3

wrr-queue cos-map 3 1 4

wrr-queue cos-map 3 2 5

priority-queue cos-map 1 6 7

channel-group 4 mode desirable

!

As you can see, on each interface on Cisco 7609, the load balance is not the best.

sh int port 4

30 second input rate 65055000 bits/sec, 15237 packets/sec

30 second output rate 109223000 bits/sec, 16393 packets/sec

sh int g9/37

30 second input rate 30557000 bits/sec, 7148 packets/sec

30 second output rate 69000 bits/sec, 31 packets/sec

sh int g9/38

30 second input rate 7176000 bits/sec, 3785 packets/sec

30 second output rate 12526000 bits/sec, 4946 packets/sec

sh int g9/39

30 second input rate 2818000 bits/sec, 744 packets/sec

30 second output rate 95810000 bits/sec, 11306 packets/sec

sh int g9/40

30 second input rate 18054000 bits/sec, 2766 packets/sec

30 second output rate 171000 bits/sec, 183 packets/sec

Can someone help me with this?

Brgds

jmayes · ‎03-04-2008

I do not have your particular platform, but, in general, my experiences with port-channels is that you do not configure the ports -- you add the configuration to the port-channel irtual interface. Have you tried removing all interface config, joining them together as a port-channel, and then applying all link config info to the port-channel interfaces?

Also, most qos settings (but not all) only apply to output from an interface queue, meaning they apply in the outbound direction only. To get qos in both directions, you'd have to set qos on the switch links to the router as well as from the router to the switch.

Joseph W. Doherty · ‎03-04-2008

Perhaps the selected channel hashing, "port-channel load-balance dst-mac" could be the issue.

If supported, try a hashing method that uses more criteria.

gsantini · ‎03-04-2008

The "port-channel load-balance dst-mac" was selected because I have the problem before the command. It was introduces trying to solve the problem. Is the same with or without this command.

Brgds

Joseph W. Doherty · ‎03-04-2008

What's the default when not specified?

Then you haven't tried any other hash methods, if supported?

gsantini · ‎03-04-2008

This is the default

sh etherchannel load-balance

EtherChannel Load-Balancing Configuration:

src-dst-ip

mpls label-ip

EtherChannel Load-Balancing Addresses Used Per-Protocol:

Non-IP: Source XOR Destination MAC address

IPv4: Source XOR Destination IP address

IPv6: Source XOR Destination IP address

MPLS: Label or IP

Brgds

Joseph W. Doherty · ‎03-04-2008

Just had a thought, you've posted one instance of a 30 second average, what does the long term packet counts look like across each channel link?

Reason I ask, channels, I believe, preserve flow sequencing by not distributing a single flow's packets across multiple links. If correct, a short duration snapshot would show varying usage of each link. Longer term, it should balance out unless you have a couple of heavy flows that move traffic between a pair of hosts that hash out the same. (The hashing, I also believe, is deterministic. New flows with the same hash parameters will always use the same path.) If the latter situation (host pairs), some hashing methods also support using IP ports for the hash. The key is finding the method that provides the best overall distribution for your traffic, but understanding that there can still be differences in link utilization. (One of the advantages of faster links vs. channel bundles.)