Solved: A couple of questions about Load Balancing.

Alen Danielyan · ‎06-24-2010

What I have read:

For routing basics:

Route Selection in Cisco Routers.pdf

Administrative Distance.pdf

Configuring Cisco Express Forwarding - CEF.pdf

For LB basics:

How Does Load Balancing Work.pdf

Load Balancing with CEF.pdf

The main ideas, as I understand:

1. As soon as a router has 2 (up to 6) routes to the same destination (host or subnet) with the same metric it begins to use Load Balancing.

2. There are 2 ways of LB: per packet and per destination. The first is clear, the second means all traffic\sessions to the same host will go via one and always the same line.

3. LB also depends on the switching mode: FastS (destination based), ProcessS (per packet) or CEF/dCEF (support both). As I understand, the difference is in the way how destination based LB work: in case of CEF mode lines are balanced not just by destination, but by the pair [source ip - dest ip], unlike FS which sends traffic from any source to the same destination via the same line.

4. Recently CEF mode is used by default on most part of routers (using last IOS versions) and by default destination based LB is activated (it is less heavy for routers CPU and memory, but potentially less effective. The latter depends on the quantity of used destinations and equality of data flows: more they are - more effective is LB).

Are these statements correct? If not, please comment mistakes.

Now my questions (the ones I remember, later I may ask some more):

1. In case of destination based LB, if we have routes to a whole subnet, and we have traffic to 2 hosts from that subnet, do we get LB, or any host within the same subnet would be assumed as the same destination?

Giuseppe Larosa · ‎06-25-2010

Hello Alen,

1) see my post about sh ip cef exact-route it demonstrates CEF is able to use load balancing towards different hosts in same FEC/IP subnet

2) exor of IP addresses less significant bits only: there is no use of Layer4 TCP or UDP ports CEF stays at OSI layer3.

3) with OSPF you can only have equal cost paths

in order to make a 2:1 ratio you need to configure the faster link as two logical links and OSPF will be able to see three equal cost paths

the logical links can be used if :

a) the interface is serial and you use frame-relay encapsulation and you define two FR subinterfaces on faster link

b) the interface is ethernet and you define two Vlan based subinterfaces

all the three logical links will need to have the same OSPF cost this can be adjusted with ip ospf cost xx or bandwidth command

if a) or b) are feasible you can use OSPF otherwise you need to move to EIGRP and to use variance with it (EIGRP specific)

Hope to help

Giuseppe

View solution in original post

Giuseppe Larosa · ‎06-27-2010

Hello Alen,

>> I have read that CEF is not applied on the encrypted traffic. Meanwhile, almost all my traffic is encrypted (as it is thraffic from branches to head office via public networks), we use DMVPN (as I understand GRE + IPSec, routing is done via OSPF over GRE).

the GRE tunnel is still an interface, I would say this is IOS version dependent as CEF support for DMVPN has been introduced at same point.

the right tool is feature navigator

http://www.cisco.com/go/fn

search by feature: DMVPN

there is a feature called

Next Hop Resolution Protocol (NHRP) - CEF rewrite for DMVPN Phase 3 Networks

Also the DMVPN solution reference design guide can be of help

http://www.cisco.com/en/US/docs/solutions/Enterprise/WAN_and_MAN/DMVPN_2_Phase2.html#wp38089

you may need to implement DMVPN phase 3 with NHRP next-hop CEF rewrite that means a different way to configure routing over the DMVPN.

in other words this is something that should be tested. In theory for the presence of the tunnel interfaces (the MGRE tunnel) there is a potential for CEF switching with the appropriate configuration (probably DMVPN phase3) and IOS images on devices.

Encryption is performed later after having chosen the exit tunnel interface.

Hope to help

Giuseppe

View solution in original post

KARUPPUCHAMY MALAIYANDI · ‎06-24-2010

Hi

//1. In case of destination based LB, if we have routes to a whole subnet, and we have traffic to 2 hosts from that subnet, do we get LB, or any host within the same subnet would be assumed as the same destination?//

It will use the same path since the destination ip will comes under the subnet.To reach the destination hosts, the router does not have a specific route(/32 subnet route).

Thanks

Karuppu

Giuseppe Larosa · ‎06-24-2010

Hello Karuppu,

see with CEF

sh ip cef exact-route 10.104.64.3 10.99.131.6

10.104.64.3 -> 10.99.131.6 => IP adj out of TenGigabitEthernet2/5, addr 10.82.2.173

sh ip cef exact-route 10.104.64.3 10.99.131.5

10.104.64.3 -> 10.99.131.5 => IP adj out of TenGigabitEthernet2/1, addr 10.82.0.162

the FEC is 10.99.131.0/24 you don't need /32 routes to take advantage of two parallel equal cost paths

This is also the reason why implementing port-channels on routers is not needed

Hope to help

Giuseppe

Ganesh Hariharan · ‎06-24-2010

What I have read:

For routing basics:

Route Selection in Cisco Routers.pdf

Administrative Distance.pdf

Configuring Cisco Express Forwarding - CEF.pdf

For LB basics:

How Does Load Balancing Work.pdf

Load Balancing with CEF.pdf

The main ideas, as I understand:

1. As soon as a router has 2 (up to 6) routes to the same destination (host or subnet) with the same metric it begins to use Load Balancing.

2. There are 2 ways of LB: per packet and per destination. The first is clear, the second means all traffic\sessions to the same host will go via one and always the same line.

3. LB also depends on the switching mode: FastS (destination based), ProcessS (per packet) or CEF/dCEF (support both). As I understand, the difference is in the way how destination based LB work: in case of CEF mode lines are balanced not just by destination, but by the pair [source ip - dest ip], unlike FS which sends traffic from any source to the same destination via the same line.

4. Recently CEF mode is used by default on most part of routers (using last IOS versions) and by default destination based LB is activated (it is less heavy for routers CPU and memory, but potentially less effective. The latter depends on the quantity of used destinations and equality of data flows: more they are - more effective is LB).

Are these statements correct? If not, please comment mistakes.

Now my questions (the ones I remember, later I may ask some more):

1. In case of destination based LB, if we have routes to a whole subnet, and we have traffic to 2 hosts from that subnet, do we get LB, or any host within the same subnet would be assumed as the same destination?

Hi,

CEF is advanced Layer 3 switching technology which can be used for load balancing in routers, By default, CEF uses per-destination load balancing. If it is enabled on an interface, per-destination load balancing forwards packets based on the path to reach the destination. If two or more parallel paths exist for a destination, CEF takes the same path (single path) and avoids the parallel paths. This is a result of the default behavior of CEF.

CEF takes the single path in cases when load sharing is done simultaneously on interfaces of different physical types, such as serial and tunnel. The hash algorithm determines the path to be chosen. In order to utilize all the parallel paths in CEF and load balance the traffic, you must enable per-packet load balancing when you have different physical interfaces like serial and tunnel. So, on the basis of the configuration and topology (serial or tunnel), load sharing can fail to work correctly with the default CEF load balancing mode.

Hope to Help !!

Ganesh.H

Alen Danielyan · ‎06-24-2010

1. In case of destination based LB, if we have routes to a whole subnet, and we have traffic to 2 hosts from that subnet, do we get LB, or any host within the same subnet would be assumed as the same destination?

After reading "Load balancing with Cisco Express Forwarding" I can say, that when (if) route-cache based forwarding mechanism is enabled (fast switching or CEF) it creates entries for each host on the destination subnet (hosts which once where "requested"), thus destination based LB is working also for the separate hosts from one subnet.

New questions:

2. It is said for CEF mode destination based LB, that it is based on [source host - destination host] pair, unlike FS which sends traffic from any source destined to the same destination via one (same) line.

My question is, does protocol or port matter, I mean when we say "source host - destination host" does it mean ip - ip, or ip:proto:port - ip:proto:port? Does the whole traffic from one ip destined to another ip go via one line even if the traffic contains multiple sessions of multiple services (from different source ports to different destination ports)?

Hope, you understand the question.

3. I have 2 OSPF created routes to the same subnet, OSPF cost is the same => LB is activated.

Now, one of my lines has 512kb/sec, another - 256kbit/sec of bandwidth. Can I organize LB proportionally to the lines bandwidth? I mean 2/1 (every 2 pairs of source-destination work via the first line, the 3-rd - via the second line, 4 and 5-th via the first line, 6-th - via the second line, and so on).

Giuseppe Larosa · ‎06-25-2010

Hello Alen,

1) see my post about sh ip cef exact-route it demonstrates CEF is able to use load balancing towards different hosts in same FEC/IP subnet

2) exor of IP addresses less significant bits only: there is no use of Layer4 TCP or UDP ports CEF stays at OSI layer3.

3) with OSPF you can only have equal cost paths

in order to make a 2:1 ratio you need to configure the faster link as two logical links and OSPF will be able to see three equal cost paths

the logical links can be used if :

a) the interface is serial and you use frame-relay encapsulation and you define two FR subinterfaces on faster link

b) the interface is ethernet and you define two Vlan based subinterfaces

all the three logical links will need to have the same OSPF cost this can be adjusted with ip ospf cost xx or bandwidth command

if a) or b) are feasible you can use OSPF otherwise you need to move to EIGRP and to use variance with it (EIGRP specific)

Hope to help

Giuseppe

Alen Danielyan · ‎06-26-2010

Dear giuslar,

Sorry I did not saw your first post. Thank you very much for the answers.

I have Ethernet connections and the variant with definition of two Vlan based subinterfaces is applicable. I am not sure I'll use it, but the info is very useful, thanks again.

A new question connected with q. 3.:

I have read that CEF is not applied on the encrypted traffic. Meanwhile, almost all my traffic is encrypted (as it is thraffic from branches to head office via public networks), we use DMVPN (as I understand GRE + IPSec, routing is done via OSPF over GRE).

Now from one side I have 2 OSPF routes with the same cost and metric:

CORE>show ip route 192.168.11.0
Routing entry for 192.168.11.0/24
Known via "ospf 1", distance 110, metric 1001, type intra area
Redistributing via ospf 2
Advertised by ospf 2 subnets match internal external 1 & 2 route-map redistr_ospf1_2_ospf2
Last update from 192.168.96.6 on Tunnel2, 1d08h ago
Routing Descriptor Blocks:
* 192.168.97.6, from 192.168.98.11, 1d15h ago, via Tunnel1
      Route metric is 1001, traffic share count is 1
    192.168.96.6, from 192.168.98.11, 1d08h ago, via Tunnel2
      Route metric is 1001, traffic share count is 1

From the other side traffic is going via (encrypted) tunnels.

My questions are:

4.1 Is LB done when PCs in the branch connect with PC (or PCs) in the head office?

4.2 Is CEF or FS used?

I see this on the core router:

CORE>show ip cef 192.168.11.0
192.168.11.0/24
nexthop 192.168.96.6 Tunnel2
nexthop 192.168.97.6 Tunnel1

And if the latter is used then (even if LB is done) all traffic from branch to File server in the head office is going via one tunnel/line?

The last question is very important, because as you can imagine there are only a few servers in the head to which branches' PCs are making requests. And destination based LB in FS variant (when only destination ip does matter, source ip - does not) could be almost useless, especially when one of my lines is 2 times slower (and it could be utilised even heavier than the faster one, thus making things even worse). => I have to:

1) provide 2:1 ratio utilisation the way you advised

2) increase slower line OSPF cost to make it backup line.

As I have almost 20 branch routers, and I am not Cisco specialist it's much eaiser to go the 2-nd way...

Giuseppe Larosa · ‎06-24-2010

Hello Alen,

>> 1. In case of destination based LB, if we have routes to a whole subnet, and we have traffic to 2 hosts from that subnet, do we get LB, or any host within the same subnet would be assumed as the same destination?

if using CEF or dCEF you may use both links or both flows use the same link

the maths operation is an exor of less significant bit of IP source address, exor with less significant bit of destination address exor with an hash seed that the router will keep until next reload

So CEF destination load balancing on equal cost paths is effective when the number of IP flows travelling over the links is great practically for Nflows > 50 you get good results.

CEF destination load balancing on equal cost paths can be not effective:

if very few flows with very high traffic volume are on the links.

Example: DB synchronization between two servers moving tens of GB it is a single flow and only one link per direction is used.

To be noted packets of a flow traveling in the opposite direction may use a different link because each router uses its own hash seed to choice the exit link.

The results of CEF load balancing algorythm can be emulated using sh ip cef exact-route

Example:

sh ip route 10.99.131.0

Routing entry for 10.99.131.0/24

Known via "isis", distance 115, metric 20020, type level-2

Redistributing via isis

Last update from 10.82.2.173 on TenGigabitEthernet2/5, 2d21h ago

Routing Descriptor Blocks:

* 10.82.2.198, from 10.80.0.166, via TenGigabitEthernet2/4

Route metric is 20020, traffic share count is 1

10.82.2.173, from 10.80.0.166, via TenGigabitEthernet2/5

Route metric is 20020, traffic share count is 1

10.82.0.162, from 10.80.0.166, via TenGigabitEthernet2/1

Route metric is 20020, traffic share count is 1

sh ip cef exact-route 10.104.64.2 10.99.131.5

10.104.64.2 -> 10.99.131.5 => IP adj out of TenGigabitEthernet2/1, addr 10.82.0.16

sh ip cef exact-route 10.104.64.3 10.99.131.5

10.104.64.3 -> 10.99.131.5 => IP adj out of TenGigabitEthernet2/1, addr 10.82.0.162

sh ip cef exact-route 10.104.64.3 10.99.131.6

10.104.64.3 -> 10.99.131.6 => IP adj out of TenGigabitEthernet2/5, addr 10.82.2.173

sh ip cef exact-route 10.104.64.3 10.99.131.5

10.104.64.3 -> 10.99.131.5 => IP adj out of TenGigabitEthernet2/1, addr 10.82.0.162

as you can see for each combination SA, DA a different link is chosen

There are very few cases when you may need per packet load-balancing that can cause issues to application by delivering out of order packets

In real world you should be fine with destination based CEF load balancing in almost all scenarios.

Hope to help

Giuseppe

Alen Danielyan · ‎06-25-2010

giuslar,

Thank you for the "sh ip cef exact-route", it was useful.

Can you answer my last questions, please?

Alen Danielyan · ‎06-25-2010

Strange, when I look on the thread it is shown the last reply was from giuslar?!

It's ok now.

Alen Danielyan · ‎06-26-2010

People! Why my new post went to the middle of the thread?

I understand it came to the post to which I reply, but how should one understand and found which posts are new ones?

I decided to repeat my last questions at the end of the thread, for people to be able to see it without looking for a new posts in the whole thread:

Dear giuslar,

Sorry I did not saw your first post. Thank you very much for the answers.

I have Ethernet connections and the variant with definition of two Vlan based subinterfaces is applicable. I am not sure I'll use it, but the info is very useful, thanks again.

A new question connected with q. 3.:

I have read that CEF is not applied on the encrypted traffic. Meanwhile, almost all my traffic is encrypted (as it is thraffic from branches to head office via public networks), we use DMVPN (as I understand GRE + IPSec, routing is done via OSPF over GRE).

Now from one side I have 2 OSPF routes with the same cost and metric:

CORE>show ip route 192.168.11.0
Routing entry for 192.168.11.0/24
Known via "ospf 1", distance 110, metric 1001, type intra area
Redistributing via ospf 2
Advertised by ospf 2 subnets match internal external 1 & 2 route-map redistr_ospf1_2_ospf2
Last update from 192.168.96.6 on Tunnel2, 1d08h ago
Routing Descriptor Blocks:
* 192.168.97.6, from 192.168.98.11, 1d15h ago, via Tunnel1
      Route metric is 1001, traffic share count is 1
    192.168.96.6, from 192.168.98.11, 1d08h ago, via Tunnel2
      Route metric is 1001, traffic share count is 1

From the other side traffic is going via (encrypted) tunnels.

My questions are:

4.1 Is LB done when PCs in the branch connect with PC (or PCs) in the head office?

4.2 Is CEF or FS used?

I see this on the core router:

CORE>show ip cef 192.168.11.0
192.168.11.0/24
nexthop 192.168.96.6 Tunnel2
nexthop 192.168.97.6 Tunnel1

And if the latter is used then (even if LB is done) all traffic from branch to File server in the head office is going via one tunnel/line?

The last question is very important, because as you can imagine there are only a few servers in the head to which branches' PCs are making requests. And destination based LB in FS variant (when only destination ip does matter, source ip - does not) could be almost useless, especially when one of my lines is 2 times slower (and it could be utilised even heavier than the faster one, thus making things even worse). => I have to:

1) provide 2:1 ratio utilisation the way you advised

2) increase slower line OSPF cost to make it backup line.

As I have almost 20 branch routers, and I am not Cisco specialist it's much eaiser to go the 2-nd way...

Giuseppe Larosa · ‎06-27-2010

Hello Alen,

>> I have read that CEF is not applied on the encrypted traffic. Meanwhile, almost all my traffic is encrypted (as it is thraffic from branches to head office via public networks), we use DMVPN (as I understand GRE + IPSec, routing is done via OSPF over GRE).

the GRE tunnel is still an interface, I would say this is IOS version dependent as CEF support for DMVPN has been introduced at same point.

the right tool is feature navigator

http://www.cisco.com/go/fn

search by feature: DMVPN

there is a feature called

Next Hop Resolution Protocol (NHRP) - CEF rewrite for DMVPN Phase 3 Networks

Also the DMVPN solution reference design guide can be of help

http://www.cisco.com/en/US/docs/solutions/Enterprise/WAN_and_MAN/DMVPN_2_Phase2.html#wp38089

you may need to implement DMVPN phase 3 with NHRP next-hop CEF rewrite that means a different way to configure routing over the DMVPN.

in other words this is something that should be tested. In theory for the presence of the tunnel interfaces (the MGRE tunnel) there is a potential for CEF switching with the appropriate configuration (probably DMVPN phase3) and IOS images on devices.

Encryption is performed later after having chosen the exit tunnel interface.

Hope to help

Giuseppe

Alen Danielyan · ‎06-28-2010

Thank you for the info, giuslar,

Unfortunately not everything you posted is clear to me ...

Anyway, I'll try to check if LB is working in my case by using the method described in "Troubleshooting Load Balancing Over
Parallel Links Using Cisco Express Forwarding" under chapter "Verifying Cisco Express Forwarding Load Balancing". Not sure that is applicable for my case, I'll report as soon as I try it (in 2 weeks).

Alen Danielyan · ‎06-28-2010

One more question:

5. I have LB via OSPF routes, I want to switch to primary/backup scheme. Should I add OSPF cost on both ends?

What happen If I change (increase) the backup line OSPF cost only on one end?

As I understand, connections initiated by the corrected end will use one line only and receive replies via the same line, but connection initiated by the non-corrected end will use both lines (according to destination based LB) and back traffic will or will not come by the backup line?

Giuseppe Larosa · ‎06-29-2010

Hello Alen,

if you are going to move to a primary/backup schema using two DMVPN clouds I suggest to move costs on both ends for consistency.

You can then move some specific traffic quotas using PBR To idle secondary VPN clouds in both sides this gives you the ability to divert traffic over secondary DMVPN cloud.

Hope to help

Giuseppe