3750 Etherchannle load-balnace Issue

Unanswered Question
Nov 1st, 2008

Hi,

I have 3750 aggregation switches as aggregation layer to connect 2 access router to Core router. 3750 runs e etherchannel from to Core with 2 1 gig link

I have src-mac based loadbalanced method used in 3750 and what I see that 3750 use only one link part of ethrchannel so during test when i have total traffic comming more than 1gig , i see drops....

Here are details :

Mac address : 000d.edac.8900 - Access Router 1

0006.d61b3c1a - Access Router r2

0015.c75d.d42c- Core router

sw1.LAB-3750G#show etherchannel summary | beg Group

Group Port-channel Protocol Ports

------+-------------+-----------+-----------------------------------------------

1 Po1(SU) - Gi2/0/1(P) Gi2/0/4(P)

sw1.LAB-3750G#

sw1.LAB-3750G#test etherchannel load-balance interface port-channel 1 mac 000d.edac.8900 0015.c75d.d42c

Would select Gi2/0/1 of Po1

sw1.LAB-3750G# test etherchannel load-balance interface port-channel 1 mac 0006.d61b.3c1a 0015.c75d.d42c

Would select Gi2/0/1 of Po1

As per above test, I see that 3750 select Gi2/0/1 for two source mac-address and that's reason i have only one link utilized..

Does some body knows the algorithm used by 3750 for above two source-mac address and how did it select always gi2/0/1 not other link for 2nd stream of traffic ??

Unforutnately I can't do src-dst-ip based loadbalnace becuase incomming traffic is MPLS labelled and existing 3750 doens't have capbility to understand MPLS so it takes as non-ip traffic..

Thanks in advance,

Chintan

I have this problem too.
0 votes
  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 5 (1 ratings)
Loading.
Giuseppe Larosa Sat, 11/01/2008 - 07:20

Hello Chintan,

the algorithm is so simple as:

exor of last bit (righmost bit) of MAC SA and MAC DA

so you have in both cases :

0 exor 0 = 0 and always the same link chosen

In general:

n bits where 2^n is M the number of links if M is even or 2^n if M is odd with 2^n = 2 *(M+1)

n=3 if M=3

n=2 if M=4

even changing to source MAC only will not change the choice

you could try to modify the mac address in one of the two routers if you can

Hope to help

Giuseppe

chintan-shah Sat, 11/01/2008 - 07:42

Hi Giuseppe,

Thanks for alogirthm.

So what you suggest, since destination mac-address is always same ( CR router mac-address) ??

I think I can change one of the access router mac-address with last digit to 1 so that 3750 choose both link each for one stream...

But in practically, I may have 4 links part of etherchannle and dst-mac address will be always same and X-OR operation may fall in to using only either of 2 link right ?

Regards,

Chintan

chintan-shah Sat, 11/01/2008 - 07:59

Hi agian,

I have load balnaced method only src-based mac address so my question is still dst-mac address play role. why does switch do XOR operation than using right most n bits

sw1.LAB-3750G#show etherchannel load-balance

EtherChannel Load-Balancing Configuration:

src-mac

EtherChannel Load-Balancing Addresses Used Per-Protocol:

Non-IP: Source MAC address

IPv4: Source MAC address

IPv6: Source MAC address

sw1.LAB-3750G#

Regards,

Chintan

Giuseppe Larosa Sat, 11/01/2008 - 08:15

Hello Chintan,

in this case DA MAC address shouldn't play any role.

the rightmost n bits of the MAC SA says which link you are going to use

but if rightmost bits are equal the same link is chosen

Hope to help

Giuseppe

chintan-shah Sat, 11/01/2008 - 08:31

Hi Giuseppe,

Now I changed src-mac for one of the access router with rightmost bit 1 but i still see on link is being used.

ar1.LAB-7609#show int Ge-WAN 2/1 | in Hardware

Hardware is GigabitEthernet Interface, address is 000d.edac.8901 (bia 000d.edac.8900)

sar1.LAB-7609#

8901 - changed the mac-address but it looks like aggregation 3750 switch still using 8900 as src how do I make sure that packet from router goes with src-mac 8901.

sw1.LAB-3750G#show mac-address-table dynamic | in Gi2/0/2

51 000d.edac.8900 DYNAMIC Gi2/0/2

51 000d.edac.8901 DYNAMIC Gi2/0/2

sw1.LAB-3750G#

Regards,

Chintan

Giuseppe Larosa Sat, 11/01/2008 - 11:36

Hello Chintan,

after 300 seconds the old entry should be removed from CAM table on C3750

So now it shouldn't appear

Hope to help

Giuseppe

chintan-shah Sun, 11/02/2008 - 01:15

Hi Giuseppe,

After 300 secs old entry disapperas and i only see new mac-address but when I sent ping from router , i agin see old and new both mac-address and I feel 3750 still using old mac for loadbalance...

sw1.LAB-3750G#show mac-address-table | in Gi2/0/2

Mac Address Table

-------------------------------------------

Vlan Mac Address Type Ports

---- ----------- -------- -----

51 000d.edac.8901 DYNAMIC Gi2/0/2

### AFTER sending PING ofr 5 packets.

sw1.LAB-3750G#show mac-address-table | in Gi2/0/2

Mac Address Table

-------------------------------------------

Vlan Mac Address Type Ports

---- ----------- -------- -----

51 000d.edac.8900 DYNAMIC Gi2/0/2

51 000d.edac.8901 DYNAMIC Gi2/0/2

53 0015.c75d.d42c DYNAMIC Po1

Total Mac Addresses for this criterion: 13

sw1.LAB-3750G#

Any cluse ?

Regards,

Chintan

Giuseppe Larosa Sun, 11/02/2008 - 01:31

Hello Chintan,

you told you are using MPLS.

Verify on the router sending the ping requests if they are sent out inside mpls frames

do this with

sh mpls forwarding

The c3750 can only see MAC addresses as source in router's frames so the only possibility is that the router is using two different source addresses for different protocols (IP and MPLS for example).

Try to shut and then unshut the router interface to give it a chance to use the new MAC address in all cases.

Hope to help

Giuseppe

chintan-shah Sun, 11/02/2008 - 01:16

Hello Giuseppe,

Yes, We are using MPLS.

I see that on router( access) it use new sourc-mac address for destination prefix/lable

sar1.LAB-7609#show mls cef adjacency mpls 16017

Index: 360448 smac: 000d.edac.8901, dmac: 0015.c75d.d42c

mtu: 1530, vlan: 4079, dindex: 0x0, l3rw_vld: 1

packets: 0, bytes: 0

Index: 491520 smac: 000d.edac.8901, dmac: 0015.c75d.d42c

mtu: 1530, vlan: 4079, dindex: 0x0, l3rw_vld: 1

packets: 0, bytes: 0

But some how 3750 still use 8900 old mac-address and it learns on mac-add table...

Regards,

Chintan

chintan-shah Sun, 11/02/2008 - 02:10

Hi Giesuppe,

I believer your theroy at some extent of using diff src-mac for diff protocol . In my case for IP and MPLS LSP.

I attach test results, let me know your thought...

Attachment: 
Giuseppe Larosa Sun, 11/02/2008 - 11:26

Hello Chintan,

your results are interesting and confirms my guess: by the way it is a smart idea that of the MAC address filter on the C3750.

first of all IS-IS is neither IP and neither MPLS it has its own PDU over ethernet.

So here you have three protocols and the change of mac-address is effective for IP and IS-IS and not for MPLS.

The interesting part is that this is a 7609 very powerful and very complex where MLS and MPLS have to coexist.

So the commands that really count should be:

show mpls forwarding 192.85.12.1 detail

notice that already in another thread under service providers/MPLS was noticed that the two commands:

sh mpls forwarding

sh mls cef

were providing different outputs and the observed behaviour was like the one showed in sh mpls forwarding (it was an issue about load-balancing).

However, you have already tried to remove the mpls from the interface.

I would try, but only if you are in a lab setup, something more invasive:

the following options the second to be used if the first doesn't fix and so on.

a)try to shut down the ge-wan 2/1 and then re-enable it.

b) disable mpls at node level, then reenable it

and you like and you can

save config with mpls active and modified mac-address.

you can try to restart the module 2

c) #hw-module module 2 reset

d) even to reload the whole system

if none of the suggested actions solves you have hit a bug that could be stated as: MPLS code uses always BIA instead of current interface MAC address.

I see also you are using the old style commands tag instead of mpls

What IOS release is in your C7609 ?

You can try to look for known bugs with bug toolkit.

Let us know if you have been able to solve this issue.

Hope to help

Giuseppe

chintan-shah Sun, 11/02/2008 - 18:14

Hi Giuseppe,

I tried option C and d but didn't help. I already tried shut and not shut of GE-WAN2/1interface too.

I could not understand option b)

disable mpls at node level, then reenable it

and you like and you can

save config with mpls active and modified mac-address.

you can try to restart the module 2

IOS:

disk0:s72033-adventerprisek9_wan-mz.122-18.SXF12a.bin

s72033-pk9sv-mz-eng-sp-122-17d.SXB11A-v3

Regards,

Chintan

chintan-shah Sun, 11/02/2008 - 20:30

Hi Giuseppe,

My all test I did with 12.2(18)SXF12a.

I have upgraded 7600 with s72033-pk9sv-mz-eng-sp-122-17d.SXB11A-v3 ( we use in network).

It works fine...

sw1.LAB-3750G#show mac-address-table | in Gi2/0/2

51 000d.edac.8901 DYNAMIC Gi2/0/2

sw1.LAB-3750G#

Regards,

Chintan

chintan-shah Mon, 11/03/2008 - 00:30

Hi Giuseppe,

Also it seems that algorithem mentioned by you don't look correct.

I have 4 link

Dst mac address right most 2 digit : 10

I use source-mac address such way that each time i have right most 2 digit diff i.e 11,01,10,00 so i should see each time diff link but for 10 and 00 i see same link...

Group Port-channel Protocol Ports

------+-------------+-----------+-----------------------------------------------

1 Po1(SU) - Gi3/0/5(P) Gi3/0/6(P) Gi3/0/8(P)

Gi3/0/9(P)

sw1#

sw1#$d-balance interface port-channel 1 mac 0011.2133.773f 0019.30f1.6b06

Would select Gi3/0/8 of Po1

sw1#$d-balance interface port-channel 1 mac 0011.2133.7731 0019.30f1.6b06

Would select Gi3/0/5 of Po1

sw1#$d-balance interface port-channel 1 mac 0011.2133.7736 0019.30f1.6b06

Would select Gi3/0/6 of Po1

sw1#$d-balance interface port-channel 1 mac 0011.2133.7730 0019.30f1.6b06

Would select Gi3/0/6 of Po1 ===>I was expecting Gi3/0/9

Am I doing soemthing wrong ?

Regards,

Chintan

chintan-shah Mon, 11/03/2008 - 04:36

Hi Giuseppe,

Yes, i don't know but upgrading IOS solved. so i can think of bug on 12.2(18)SXF12a

Regarding alogrimt, if you look my test for four diff src-mac such way that each time i should have diff link. I don't get considering algorithm explained by you...

Can you give me example ? I am still not confident on algorithm after doing test ...

Regards,

Chintan

chintan-shah Tue, 12/02/2008 - 09:54

Hi Giuseppe,

With this issue, I am thinking to put 6500 than 3750 as to my knowledge, 6500 support MPLS so when packets comes on layer 2 port , 6500 can look MPLS and in turn IP so can do load-balance on ip flow based....

I will be using 6500 layer 2 device like 3750...

Just want to confirm that as layer 2 6500 have capability to look MPLS so can solve my purpose to look IP header and even MPLS label based load balancing....

Regards,

Chintan

Giuseppe Larosa Tue, 12/02/2008 - 12:21

Hello Chintan,

I'm looking at configuration guide for 12.2(33)SXH release.

about load-balancing says:

The selected mode applies to all EtherChannels configured on the switch. EtherChannel load balancing can use MPLS Layer 2 information.

see

http://www.cisco.com/en/US/docs/switches/lan/catalyst6500/ios/12.2SX/configuration/guide/channel.html#wp1020414

and later about the load balancing method command:

about the various methods available:

•mpls-Load balancing for MPLS packets

•src-dst-ip-Source and destination IP addresses

MPLS is listed as a different option here.

Unfortunately, I cannot reach the C6500 command reference to see if there is some additional info.

Some considerations: if MPLS method is supported it is something different like load balancing based on MPLS label value (topmost label).

I don't think MPLS load-balancing means the device will look inside the MPLS frames and performs src-dest-ip or other type.

This would cause additional work for the switch that can be a problem.

However, you can think to try if you know that from an MPLS point of view (MPLS labels used) the traffic has enough variations.

Being L2 switch should'nt be an issue if PFC supports MPLS (sup 720 3B or better).

Another question is if you have a mix of IP and MPLS traffic to be sent over the bundle.

I would suggest to perform some testings in lab if you can to see if this move can provide you a real advantage.

Hope to help

Giuseppe

chintan-shah Tue, 12/02/2008 - 19:04

Hi Giuseppe,

I agree that i have to test in lab to see thre is not impact. But what I see here one more option , mpls label-ip....can't get what would be benefit ?

sw1.LAB(config)#port-channel load-balance mpls label-ip

sw2.LAB(config)#end

sw2.LAB#show etherchannel load-balance

EtherChannel Load-Balancing Configuration:

src-dst-ip

mpls label-ip

EtherChannel Load-Balancing Addresses Used Per-Protocol:

Non-IP: Source XOR Destination MAC address

IPv4: Source XOR Destination IP address

IPv6: Source XOR Destination IP address

MPLS: Label or IP

Giuseppe Larosa Tue, 12/02/2008 - 22:08

Hello Chintan,

yes there are two possible options:

RT-RM-SXR000-D-E-1(config)#port-channel load-balance mpls ?

label Use MPLS label only

label-ip Use MPLS label or IP

however, I think the latter is for a mix of IPv4 and MPLS traffic so that it can try to load balance both types of traffic.

Again, I don't think this means the switch will look at ip header inside an MPLS frame: ads you see it says:

label-ip Use MPLS label or IP

not MPLS label AND IP

Edit:

I see from your show command that is able to use different methods for different types of traffic so it looks like that for MPLS it can access the ip header to make a load balancing decision.

Edit2:

in my device that is a PE the option is already active without explicit configuration:

RT-RM-SXR000-D-E-1#sh etherchannel load-balance

EtherChannel Load-Balancing Configuration:

src-dst-mac

mpls label-ip

EtherChannel Load-Balancing Addresses Used Per-Protocol:

Non-IP: Source XOR Destination MAC address

IPv4: Source XOR Destination MAC address

IPv6: Source XOR Destination MAC address (routed packets)

Source XOR Destination IP address (bridged packets)

MPLS: Label or IP

RT-RM-SXR000-D-E-1#

RT-RM-SXR000-D-E-1#sh run | inc port-channel load

port-channel load-balance src-dst-mac

RT-RM-SXR000-D-E-1#

But I haven't MPLS traffic over the bundle in normal conditions.

Hope to help

Giuseppe

Actions

This Discussion