Load balancing using Performance Routing pfr/OER

Document

Wed, 12/23/2015 - 13:06
Dec 25th, 2009
User Badges:
  • Purple, 4500 points or more
  • Community Spotlight Award,

    Best Publication, December 2015

Introduction:

Cisco Performance Routing pfr is one of the most intelligent Cisco IOS services that can handle traffic routing automatically to achieve the most reliable and none stop traffic forwarding between sites and over multiple routers and links.

Cisco pfr optimizes routing and route selection based on real time measurements of the available paths and select the best path with regard to the defined polices, such as traffic delay, jitter or link utilization, which make it more flexible and convenient especially with the implementations of converged networks (Voice, Video and DATA).

Using only a standard dynamic routing protocol such as BGP with two ISPs, if one of the ISPs experiencing problems inside the SP network and a company has real time sensitive traffic this will make some issues such delay, jitter and/or packet loss, while from BGP perspective the other BGP peer is reachable and the BGP session is up.

With Cisco pfr the router now will be able to measure the traffic over all the available ISPs/paths, this measurement will be done by the edge routers that taking the role of a border router BR in pfr terminologies, while another router (either dedicated or co-existed in one of the BRs) will be the decision maker and all the BR will report the traffic measurements to it and this router called Master Controller MC, which works exactly as the brain of the pfr.

If any ISP link experiencing any problem such delay or jitter the BR connected to that ISP will report the traffic measurement to the MC and the MC will compare it to a predefined policy, if its considered out of policy OOP then the MC will start looking for another external link within the BRs that has better or in policy path, and there are several timers that can be configured and tuned such as bakoff timer to avoid route flapping and periodic interval which is the periodic time in minutes that the MC router start learn prefixes with a default value of 120 minute.


For more details please refer to the following link:


Performance Routing Q&A

http://www.cisco.com/en/US/prod/collateral/iosswrel/ps6537/ps6554/ps6599/ps8787/prod_qas0900aecd806c4f03.html




Configuration Example:

In this configuration example we will see how we can configure load balancing by using pfr in one edge router this edge router has two external links represent WAN or Internet links ( both are valid options) and this edge router configured as BR router and MC router.

the routing configuration is very simple only two defual routes each point to one of the ISP’s next hop IP.


ip route 0.0.0.0 0.0.0.0 10.1.1.10

ip route 0.0.0.0 0.0.0.0 20.1.1.10




EDGE_RTR#show ip route


Gateway of last resort is 20.1.1.10 to network 0.0.0.0


     1.0.0.0/24 is subnetted, 1 subnets

C       1.1.1.0 is directly connected, Loopback0

     20.0.0.0/24 is subnetted, 1 subnets

C       20.1.1.0 is directly connected, FastEthernet1/1

     10.0.0.0/24 is subnetted, 1 subnets

C       10.1.1.0 is directly connected, FastEthernet1/0

C    192.168.1.0/24 is directly connected, FastEthernet2/0

S*   0.0.0.0/0 [1/0] via 20.1.1.10

                      [1/0] via 10.1.1.10


pfr.jpg

The criteria will be used here to do load balancing is link utilization, first we need to define the border router and basic MC configurations



interface Loopback0

ip address 1.1.1.1 255.255.255.0


key chain OER ---- for authentication

key 1

  key-string oerkey



MC part:


oer master

max-range-utilization percent 2     ---- 2% only for this example ( if the range between external links utilization over 2 % then the MC will start to distribute the load between the links with regard to the max link utilization configured bellow as will )

logging

!

border 1.1.1.1 key-chain OER       ----- local loopback as both BR and MC co-existed

  interface FastEthernet1/1 external

   max-xmit-utilization absolute 5       --- this means max utilization for this link 5 K ( just for the purpose of this example to see the link out of policy quickly )

  interface FastEthernet1/0 external

   max-xmit-utilization absolute 50     --- 50k ( this value for this example only )

  interface FastEthernet2/0 internal

!

learn

  throughput

  periodic-interval 0     --- configured 0 to make sure all the time the router will learn prefixes ( for the purpose of this example only)

  aggregation-type prefix-length 32   -- automatic aggregation of the prefixes that will be created by the MC automatically will have a prefix length with /32 in this example

mode route control

mode route metric static tag 2000   --- automatic static route entries created by the MC will have route tag as 2000

mode select-exit best    --- always select the best in policy exit

resolve range priority 1    --- policy measurement criteria will give utilization range priority 1

resolve utilization priority 2 variance 1 ---  link utilization priority 2



BR part:

oer border

local Loopback0      -- same ip address used in the MC part config for this BR

master 1.1.1.1 key-chain OER

!


EDGE_RTR#show oer master border

Border           Status   UP/DOWN             AuthFail

1.1.1.1          ACTIVE   UP       01:09:11          0



as it shown earlier in this document the routing table has only two static default routes each one points to a different ISP/ next hop


Now lets generate traffic from the inside network to simulate internal traffic and as we configured the link utilization of interface fa1/1 to a low value this interface will  be considered OOP quickly



EDGE_RTR#show oer master prefix



EDGE_RTR#



: %OER_MC-5-NOTICE: Load OOP BR 1.1.1.1, i/f Fa1/1,  load 24

policy 5

%OER_MC-5-NOTICE: Exit 1.1.1.1 intf Fa1/1 OOP, Tx BW 24, R

x BW 24, Tx Load 0, Rx Load 0






EDGE_RTR#show oer master prefix


Prefix                  State     Time Curr BR         CurrI/F         Protocol

                      PasSDly  PasLDly   PasSUn   PasLUn  PasSLos  PasLLos

                      ActSDly  ActLDly   ActSUn   ActLUn      EBw      IBw

--------------------------------------------------------------------------------

100.100.100.100/32   DEFAULT*      @29 1.1.1.1         Fa1/1           U

                            U        U        0        0        0        0

                            U        U        0        0       25       25

EDGE_RTR#




%OER_MC-5-NOTICE: Discovered Exit for prefix 100.100.100.100/32, BR 1.1.1.1, i/f Fa1/1




EDGE_RTR#show oer border routes static



Flags Network            Parent             Tag

CE    100.100.100.100/32 0.0.0.0/0          2000



EDGE_RTR#show ip route 100.100.100.100

Routing entry for 100.100.100.100/32

  Known via "static", distance 1, metric 0

  Tag 2000

  Routing Descriptor Blocks:

  * 10.1.1.10

      Route metric is 0, traffic share count is 1

      Route tag 2000



As it shown above a static route entry has been created automatically with a prefix length of /32 based on the prefix aggregation length specified in the MC config

Also the route tag is 2000 which is the tag value configured in the MC config as well, this tag maybe useful for redistribution or route filtering

Also the most important thing is that the created static route entry for the prefix 100.100.100.100/32 is through the Fa1/0

Because the first used link was fa1/1 and according to the previous logging message this link flooded with traffic and considered out of policy


: %OER_MC-5-NOTICE: Load OOPBR 1.1.1.1, i/f Fa1/1,  load 24

policy 5



then the MC has chosen fa1/0 because it is not over utilized ( in policy )


now lets generate traffic again to over utilize fa1/0 :




%OER_MC-5-NOTICE: Load OOPBR 1.1.1.1, i/f Fa1/0,  load 53

policy 50

%OER_MC-5-NOTICE: Exit 1.1.1.1 intf Fa1/0 OOP, Tx BW 53, R

x BW 53, Tx Load 0, Rx Load 0




New learned prefixes:




Prefix                  State     Time Curr BR         CurrI/F         Protocol

                      PasSDly  PasLDly   PasSUn   PasLUn  PasSLos  PasLLos

                      ActSDly  ActLDly   ActSUn   ActLUn      EBw      IBw

--------------------------------------------------------------------------------

200.200.200.200/32   DEFAULT*      @74 1.1.1.1         Fa1/0           U

                            U        U        0        0        0        0

                           U        U        0        0       29       28

100.100.100.100/32   INPOLICY        0 1.1.1.1         Fa1/0           STATIC

                            U        U        0        0        0        0

                            U        U        0        0        0        0





%OER_MC-5-NOTICE: Discovered Exit for prefix 200.200.200.200/32, BR 1.1.1.1, i/f Fa1/0




EDGE_RTR#show oer border routes static


Flags: C - Controlled by oer, X - Path is excluded from control,

       E - The control is exact, N - The control is non-exact


Flags Network            Parent             Tag

CE    100.100.100.100/32 0.0.0.0/0          2000

CE    200.200.200.200/32 0.0.0.0/0          2000


EDGE_RTR#show ip route


     200.200.200.0/32 is subnetted, 1 subnets

S       200.200.200.200 [1/0] via 20.1.1.10

     1.0.0.0/24 is subnetted, 1 subnets

C       1.1.1.0 is directly connected, Loopback0

     100.0.0.0/32 is subnetted, 1 subnets

S       100.100.100.100 [1/0] via 10.1.1.10

     20.0.0.0/24 is subnetted, 1 subnets

C       20.1.1.0 is directly connected, FastEthernet1/1

     10.0.0.0/24 is subnetted, 1 subnets

C       10.1.1.0 is directly connected, FastEthernet1/0

C    192.168.1.0/24 is directly connected, FastEthernet2/0

S*   0.0.0.0/0 [1/0] via 20.1.1.10

                    [1/0] via 10.1.1.10



It obvious from the above routing table we have two new static routes entries crated automatically by the MC each one point to a different next hop based on the link's utilization policy configuration. Here we achieved load balancing over tow links by using pfr.

.

Note:

You can implement NAT with this solution if its required by using ACLs and route maps

This example was configured and working with NAT:




EDGE_RTR#show ip nat translations

Pro Inside global      Inside local       Outside local      Outside global

icmp 20.1.1.1:84       192.168.1.1:84     200.200.200.200:84 200.200.200.200:84

icmp 10.1.1.1:85       192.168.1.1:85     100.100.100.100:85 100.100.100.100:85

EDGE_RTR#




For a configuration example of using NATing with two links and route maps please see the document bellow:

https://supportforums.cisco.com/docs/DOC-8313



This was a simple pfr configuration example; with pfr you can configure more complex policies and measurements with active and/or passive monitoring using echo, tcp or udp props in conjunction with ip sla, also you can use a dynamic routing protocol instead of static routing such as BGP.




Thank you

Marwan Alshawi

Loading.
snarayanaraju Fri, 12/25/2009 - 03:20
User Badges:

Hi  Marwan Alshawi

One of the very good article with simple example I have seen. Thanks for your contribution.

It is stored on by study material with you name. Thanks again

sairam

Marwan ALshawi Fri, 12/25/2009 - 04:04
User Badges:
  • Purple, 4500 points or more
  • Community Spotlight Award,

    Best Publication, December 2015

thank you sairam for your nice comment

tumarha Fri, 06/18/2010 - 11:10
User Badges:

Dear Sir


If I apply PBR for force some source to exit some external WAN and also enable OER, which one will be prefer?


Best regard

Wisit

Marwan ALshawi Fri, 06/18/2010 - 16:21
User Badges:
  • Purple, 4500 points or more
  • Community Spotlight Award,

    Best Publication, December 2015

OER normally controling your routing table while PBR baypass the routing table becuase it make the forwarding before the traffic get forwarded based on the routing table

in this case traffic matched by your PBR will be using PBR not OER

amilkar-loyo Mon, 09/27/2010 - 12:44
User Badges:

marwanshawi

I have a doubt regarding PFR at documentation i found MC and BR could coexist on the same router, I ha ve a customer who is asking about to add voice gateway on the same box, is it possible? does the 2911 hardware support this configuration? PFR is capable to work with packets generated on the same router?

cgm Tue, 01/25/2011 - 07:45
User Badges:

It seems OER is not that intelligent...

In your example, OER did reroute a 50k+ flow because it was overloading an exit,

to a 5k max exit ? Ouch.

Don't take me wrong. Your config is way extreme in some aspects, and this

may cause non obvious behaviours. IMHO.

Marwan ALshawi Tue, 01/25/2011 - 15:42
User Badges:
  • Purple, 4500 points or more
  • Community Spotlight Award,

    Best Publication, December 2015

As mentioned above, all of the numbers and values used are just for the purpose of the example

In a production network you need to consider each link load and bandwidth in your calculations

cgm Thu, 01/27/2011 - 03:05
User Badges:

But what if your numbers are those numbers ? What is going to prevent PfR from trying to put a 50k flow into a 5k pipe ? And then, what is going to prevent the flapping that this is going to cause ?

Performance routing is a tricky business. It's been dormant in IGRP/EIGRP for ages somehow, and nobody uses it in part for this, AFAIK. What countermeasures does PfR have to manage this ?

Marwan ALshawi Thu, 01/27/2011 - 13:47
User Badges:
  • Purple, 4500 points or more
  • Community Spotlight Award,

    Best Publication, December 2015

ok in this case you can use PfR timers such as backoff which you can tunne it to avoid flapping

by the way this article just discussing this interesting IOS feature but it dose not mean you can not use other methods or products to do loadbalncing


Regards,

sanaqvi Tue, 01/03/2012 - 10:18
User Badges:

Hello marwanshawi.  Thank you for the excellent article.  Reading the Cisco documentation would only make one dizzy, while your explanation thankfully simplifies stuff.


Could you possibly please post the complete config - including the NATing with dual links and everything?  I am able to get the dual-nat working successfully, but having issues getting it working with the OER configuration above.  May be I am missing something.  Thanks.

burhanahmed Tue, 01/29/2013 - 03:12
User Badges:

Dear Marwan shawi

what will be the configs if we have two sperate routers connectinf with two ISPs, One is primary and other is secondary and BGP is running and Primary ISP is prefered via BGP. How we can load Balance the traffic ?

concentrix-mis Mon, 11/04/2013 - 22:00
User Badges:

Hi there,



What if the exit link is intermittent? Woud you consider adding "resolve loss"???

rstalin25 Mon, 01/27/2014 - 01:21
User Badges:

Hi Marwan,


We have a Multihomed BGP, Two routers with two different service provider. Primary router will act as a master and secondary router is acting as the border. Our lan pool is advertised over both the Service provider. Based on the PFR/OER decision forward will be through service provider B, but reverse is still getting on Service provider A. Where most of the Citrix application is not working. Is there any other Way to controll the reverse traffic over BCG using PFR.



ISP 1                                           ISP2


Router 1(MC)                          Router 2 (BC)


                    LAN network (Network Range advertised on both the Router using Multihomed BGP).

rstalin25 Mon, 01/27/2014 - 01:23
User Badges:

We too tried with Dual NATTING at the edge router and found to be working. We are seeking for the root cause via PFR/OER.

expertadvisor20151 Sun, 01/25/2015 - 05:10
User Badges:

Please see Cisco's new solution for load-balancing:

ITD: Load Balancing, Traffic Steering & Clustering using Nexus 5k/6k/7k

ITD (Intelligent Traffic Director) is a hardware based multi-Tbps Layer 4 load-balancing, traffic steering and clustering solution on Nexus 5k/6k/7k series of switches. It supports IP-stickiness, resiliency, NAT (EFT), VIP, health monitoring, sophisticated failure handling policies, N+M redundancy, IPv4, IPv6, VRF, weighted load-balancing, bi-directional flow-coherency, and IPSLA probes including DNS. There is no service module or external appliance needed. ITD provides order of magnitude CAPEX and OPEX savings for the customers. ITD is available on Nexus series of switches. ITD is much superior than legacy solutions like PBR, WCCP, ECMP, port-channel, layer-4 load-balancer appliances.

Actions

This Document