Solved: Re: BGP & Multihomed Router Problem

jeff · ‎12-01-2003

We are currently having a problem with our outbound traffic. Our router is a 7206 with 3 T1s to one provider and 3 to another. We have our own registered IP class and AS number and our providers are only sending us local routes through BGP. We do have CEF enabled on the router.

Our goal with the two separate carriers was to have the redundancy if one has a failure, but we would normaly route traffic over both. When we do a traceroute to let's say sprint.com, I notice that the traceroutes will go out to both carriers simultaneously, which is causing a delay in our routing and unnecessary traffic on the circuits to both carriers.

As you can see from our router configuration, that our carriers don't have a similiar configuration. What do I need to do to balance the traffic? I'm not looking for a perfect 50/50 but I'd like to stop the traceroutes (or traffic) from going down both carriers and to balance our outbound traffic.

Thanks for any help.

Jeff

interface Loopback ProviderA

description Loopback for provideA BGP

ip address aa.bb.cc.135 255.255.255.255

!

interface Multilink1

description ****Load Balance Virtual Interface for providerB***

ip address zz.yy.xx.138 255.255.255.252

ip access-group 198 in

no cdp enable

ppp multilink

no ppp multilink fragmentation

multilink-group 1

!

interface FastEthernet0/0

description ***LAN backbone ***

ip address fff.fff.0.6 255.255.252.0

no ip route-cache

no ip mroute-cache

full-duplex

!

interface Serial5/0

description ***ProviderA***

ip address aa.bb.cc.58 255.255.255.252

ip access-group 198 in

ip load-sharing per-packet

encapsulation ppp

ip route-cache flow

no ip mroute-cache

serial restart-delay 0

no cdp enable

!

interface Serial5/1

description ***providerA***

ip address aa.bb.cc.62 255.255.255.252

ip access-group 198 in

ip load-sharing per-packet

encapsulation ppp

ip route-cache flow

no ip mroute-cache

serial restart-delay 0

no cdp enable

!

interface Serial5/2

description ***providerA***

ip address aa.bb.cc.66 255.255.255.252

ip access-group 198 in

ip load-sharing per-packet

encapsulation ppp

ip route-cache flow

no ip mroute-cache

serial restart-delay 0

no cdp enable

!

interface Serial5/3

no ip address

ip load-sharing per-packet

no ip mroute-cache

shutdown

serial restart-delay 0

no cdp enable

!

interface Serial5/4

description ***providerB***

no ip address

encapsulation ppp

no fair-queue

serial restart-delay 0

no cdp enable

ppp multilink

multilink-group 1

!

interface Serial5/5

description ***providerB***

no ip address

encapsulation ppp

no fair-queue

serial restart-delay 0

no cdp enable

ppp multilink

multilink-group 1

!

interface Serial5/6

description ***providerB***

no ip address

encapsulation ppp

no fair-queue

serial restart-delay 0

no cdp enable

ppp multilink

multilink-group 1

!

interface Serial5/7

no ip address

no ip mroute-cache

shutdown

serial restart-delay 0

no cdp enable

!

router bgp Public

no synchronization

bgp log-neighbor-changes

bgp dampening

network fff.fff.0.0 mask 255.255.192.0

neighbor zz.yy.xx.137 remote-as providerb

neighbor zz.yy.xx.137 route-map ProviderB-in in

neighbor zz.yy.xx.137 route-map local-out out

neighbor aa.bb.cc.134 remote-as ProviderA

neighbor aa.bb.cc.134 ebgp-multihop 5

neighbor aa.bb.cc.134 update-source LoopbackproviderA

neighbor aa.bb.cc.134 route-map ProviderA-in in

neighbor aa.bb.cc.134 route-map local-out out

no auto-summary

!

ip route 0.0.0.0 0.0.0.0 Multilink1

ip route 0.0.0.0 0.0.0.0 aa.bb.cc.57

ip route 0.0.0.0 0.0.0.0 aa.bb.cc.61

ip route 0.0.0.0 0.0.0.0 aa.bb.cc.65

ip route aaa.bbb.ccc.134 255.255.255.255 Serial5/0

ip route aaa.bbb.ccc.134 255.255.255.255 Serial5/1

ip route aaa.bbb.ccc.134 255.255.255.255 Serial5/2

ip as-path access-list 1 permit ^1_[0-9]*$

ip as-path access-list 2 permit ^16631_[0-9]*$

ip as-path access-list 10 permit ^$

route-map ProviderA-in permit 10

match as-path 2

route-map local-out permit 10

match as-path 10

route-map ProviderB-in permit 10

match as-path 1

ruwhite · ‎12-03-2003

Yes, you can do what's described in this tech note. If you are already getting just partials from your SP, though, the outbound filter is the main one you should be worried about, rather than the inbound one. You can use the inbound filters to reduce what your provider is sending you, though.

:-)

Russ.W

View solution in original post

ruwhite · ‎12-01-2003

It's because you're running per packet load sharing. The traceroutes aren't being duplicated, but when you traceroute, normally the software generates three to five packets towards the destination, with the TTL set to time out at each possible hop.

So, for instance, suppose you traceroute to 10.1.1.1. The traceroute software will generate, say, three packets to 10.1.1.1 with a TTL of 1, then three more with a TTL of 2, then 3 more with a TTL of 3, etc., until the TTL reaches 16. This is done so each router along the way will see a packet with a TTL expiring, and send back an ICMP notification. The traceroute software can use these ICMP notifications to figure out each router the packets cross when travelling towards the destination.

What's happening here is that the first packet from the set is reaching the edge of your network, and taking the path through one provider, the second is taking the path through the other provider, the third is taking the path through the first provider again, etc. This makes it look like you are duplicating packets, but you're not.

If you don't want to see this, and if you really want better efficiency (rather than just nicer looking serial line utilization numbers), use the normal CEF load sharing algorithm, rather than the per packet algorithm. The normal algorithm will give you pretty good load sharing without out of order packets--and out of order packets will destroy the performance of most applications communicating across a network.

:-)

Russ.W

s-almeida · ‎12-01-2003

You can do policy based routing for outgoing traffic depending upon AS's. For example the AS starting with 1 to 4 with ISP A and AS starting with 5 to 9 with ISP B.

ipotts · ‎12-02-2003

Unfortunately traceroute is not an accurate way to test the path used when you have equal cost paths since it does not use CEF, but goes via the IP input process switched queue, and will therefore always be per packet load shared, and not a source hash and destination based like CEF. I agree you should disable ip per packet loadsharing on each interface, but this will not change the output of your traceroute, however it will change the path of your actual traffic,

ruwhite · ‎12-02-2003

How would you do this? The AS Path isn't in the routing table (?), nor in the IP Packet, so I don't know how you'd match on these for policy based routing? Or do you mean route filtering?

Russ.W

preddyi · ‎12-02-2003

If your router has enough memory,

Simply ask your both the ISPs to send all internet routes to your router.

Let CEF be there as it is.

Let your router decide which ISP path to take, based on BGP best path decision.

Incase if one of your ISP links are getting choked for the out going traffic , to distribute the traffic more eventually configure your inbound rout filter to accept cetain no of AS routes via BGP neigbour A and remaing via the other BGP neigbour B.

Then for achieving redundancy accept the same recieved routes of neighbour A via neigbour B also with AS path prepended and vice-versa.

jeff · ‎12-02-2003

Thanks for all the replies. I will disable the per packet load sharing and let you know how it goes. We did have both providers sending us full routes, but the prefix table from each was over 117K. Our memory usage for this was over 72 Mb and our router memory was getting low and traffic was sluggish.

I would like to know more about the inbound route filters for the AS routes as mentioned in the couple of posts. Can you give me an example of this and possibly a link that I can read regarding this method.

Thanks again.

Jeff

preddyi · ‎12-02-2003

Pls refer,

http://www.cisco.com/en/US/tech/tk365/tk80/tech_configuration_examples_list.html

http://www.cisco.com/en/US/tech/tk365/tk80/technologies_configuration_example09186a00800945bf.shtml

http://www.cisco.com/en/US/tech/tk365/tk80/tech_tech_notes_list.html

ruwhite · ‎12-02-2003

Another option is to accept partials--tell your ISP you only want routes they are originating within their AS, the routes they are learning from their directly connected customers (not peers), and a default. That should give you fairly good optimal routing, plus complete reachability. I can't say what it would do to your load sharing, though, definitely, it's going to depend on who your two providers are.

:-)

Russ.W

jeff · ‎12-02-2003

Russ,

Thanks for the info. We are only receiving our providers routes within their AS just as you said. I did remove the per-packet statements and had the provider do the same. So far it looks good but I'm going to wait a few days to see how things settle.

Did you mention about doing the scenario mentioned in the tech document below:

http://www.cisco.com/en/US/tech/tk365/tk80/technologies_configuration_example09186a008009456d.shtml

If so, I'll make it the next step.

Thanks again.

Jeff

ruwhite · ‎12-03-2003

Yes, you can do what's described in this tech note. If you are already getting just partials from your SP, though, the outbound filter is the main one you should be worried about, rather than the inbound one. You can use the inbound filters to reduce what your provider is sending you, though.

:-)

Russ.W

jeff · ‎12-04-2003

So far so good, but I will keep an eye on things. Thanks to everyone for the replies and thanks Russ for your help. I will keep the suggested configuration handy if I need to go that route (no pun intended).

Thanks again.

Jeff