CCIE-Type Routing Question

Unanswered Question
Mar 10th, 2009
User Badges:
  • Blue, 1500 points or more

Please view the attached drawing before you read on and you may want to leave it open to refer to it.


Anyway, here is the scoop.


The OSPF area represents our Australia network.


Routers SING and JAPAN are OSPF ASBRs that redistribute OSPF-learned routes into BGP.


Both the SING and JAPAN routers form iBGP neighborships with both FMC and ECC in the United States.


So, FMC learns OSPF Australia routes through iBGP from both SING and JAPAN.


And ECC learns OSPF Australia routes through iBGP from both SING and JAPAN, too.


Right now, both FMC and ECC routers favor the SING router as the "next hop."


The reason is that all things being equal, the SING router carries a better IGP metric into its BGP advertisements than the JAPAN router does.


So, if one does a "sh ip bgp 19.210.108.16" (this is an Australia network) in the FMC and ECC routers, they will both show that they have learned 2 routes through iBGP, one from SING and one from JAPAN, but the SING router carries an IGP (OSPF) metric of, say, 330, while the JAPAN router advertises an IGP metric of 445. Therefore, the SING route is favored, so the FMC and ECC routers place the SING-learned iBGP routes in their BGP table.


<i>ecc#sh ip bgp 19.210.108.16

BGP routing table entry for 19.210.108.16/29, version 31730939

Paths: (2 available, best #1, table Default-IP-Routing-Table)

Advertised to update-groups:

2 3

Local

19.213.1.130 (metric 1466112) from 19.213.1.130 (19.213.1.130)

Origin incomplete,


  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 4.8 (8 ratings)
Loading.
lamav Tue, 03/10/2009 - 21:13
User Badges:
  • Blue, 1500 points or more

By the way, Im sorry for this long post. I tried to make it as straight forward as possible, so as not to be tedious.


Let me thank everyone in advance for taking the time to read this over and offering solutions, suggestions and comments. I appreciate it.


Victor

lamav Tue, 03/10/2009 - 21:20
User Badges:
  • Blue, 1500 points or more

I just remembered that some people may not have Visio (Jon!), so I am providing the drawing as a JPEG, too.


Thanks



Giuseppe Larosa Wed, 03/11/2009 - 02:16
User Badges:
  • Super Silver, 17500 points or more
  • Hall of Fame,

    Founding Member

Hello Victor,


the BGP metric comes from the OSPF metric if you have access to the routers you should verify the settings on remote routers ahors03 and arcr01.

you can check with

sh ip ospf interface on lan and serial interfaces on all four routers singh, japan, ahors03, arcr01


However, you can use local-preference in iBGP to give preference to japan routes


you can do this in a route-map applied outbond on japan itself or with route-maps applied inbound on FMC and ECC.


the attempt the customer is doing is that of changing the IGP metric to next hop by changing delay parameters to change the metric to the BGP next-hop.


lowest MED (metric) is preferred in BGP.


However, if we look at the path selection criteria


see


http://www.cisco.com/en/US/tech/tk365/technologies_tech_note09186a0080094431.shtml


we can see that IGP metric to next hop is considered only after a tie in MED.


so here you can do the following:

use local-preference that is checked first then MED and so you can influence the choices regardless of MED values


and/or

try to fix the different MED metrics seen on singh and japan


(this can come from different settings on remote routers being OSPF metric cumulative, a different bandwidth on serial links, and/or on lan interface on remote routers can explain a difference in metrics, a different auto-reference bandwidth in the ospf process)


So the customer's idea to modify IGP metric to next hop shouldn't work unless MED are made equal before.


I would use local-preference outbound japan router (you do it once and is spread in all the AS) but I would also check all the remote routers to understand the origin of the different OSPF metrics as seen by singh and japan.


Hope to help

Giuseppe


lamav Wed, 03/11/2009 - 07:03
User Badges:
  • Blue, 1500 points or more

Hi, Giuseppe:


Im not sure why you are discussing MEDs when they are not being utilized in this set up at all.


I agree that the IGP metric (not the same as "MED", by the way) is the tie breaker here and thats why SING is being preferred over JAPAN. I mention that in my original post.


I also agree with you that the client is trying to influence the eigrp path selection process to the BGP next hop (SING) by adjusting the delay. That was my suspicion, but I was a bit confused because I hadnt spoken to him -- still havent, actually.


What I am looking for are people's opinions on how they would handle the requirement to have Australia-destined traffic go through JAPAN, instead of SING. And you did that! Thanks


Anyone else?


Id love to hear from others.


Thank you





Giuseppe Larosa Wed, 03/11/2009 - 07:25
User Badges:
  • Super Silver, 17500 points or more
  • Hall of Fame,

    Founding Member

Hello Victor,

thanks for your kind rating


>> Im not sure why you are discussing MEDs when they are not being utilized in this set up at all.


MED = BGP metric


Let's examine the sh ip bgp


ecc#sh ip bgp 19.210.108.16

BGP routing table entry for 19.210.108.16/29, version 31730939

Paths: (2 available, best #1, table Default-IP-Routing-Table)

Advertised to update-groups:

2 3

Local

19.213.1.130 (metric 1466112) from 19.213.1.130 (19.213.1.130)

>> Origin incomplete, metric 255, localpref 100, valid, internal, best

Local

19.213.1.154 (metric 1465856) from 19.213.1.154 (19.213.1.154)

>>Origin incomplete, metric 404, localpref 100, valid, internal

ecc#


here 255 and 404 are BGP metric values also known as MED


notice that IGP metric to next hops are the values in (..)


19.213.1.130 (metric 1466112)


19.213.1.154 (metric 1465856)


and that IGP metric is already greater for 19.213.1.130 and that the BGP best path choice looks at BGP metric lowest metric first as in the link I've provided


Hope to help

Giuseppe

lamav Wed, 03/11/2009 - 08:21
User Badges:
  • Blue, 1500 points or more

Giuseppe:


I am going to have to differ with something you're saying.


The metric you site in the output of the "sh ip bgp" command on the ECC router as being the "BGP MED" (metric 255 and metric 404) are in fact the IGP metrics that BGP carried in its advertisements to FMC and ECC. The IGP in question is OSPF and the metrics are 255 and 404 to the destination network in Australia from SING and JAPAN, respectively. See below the output from the SING and JAPAN routers.


Also, the metric in the "( )" in the "sh ip bgp" command on the ECC router is the IGP metric within the AS for the route to SING and JAPAN. The IGP in this case is eigrp and the eigrp metric to get from ECC to SING is 1466112 and the eigrp metric to get from ECC to JAPAN is 1465856.


HTH


Victor




ecc#sh ip bgp 19.210.108.16

BGP routing table entry for 19.210.108.16/29, version 31730939

Paths: (2 available, best #1, table Default-IP-Routing-Table)

Advertised to update-groups:

2 3

Local

19.213.1.130 (metric 1466112) from 19.213.1.130 (19.213.1.130)

Origin incomplete, metric 255, localpref 100, valid, internal, best

Local

19.213.1.154 (metric 1465856) from 19.213.1.154 (19.213.1.154)

Origin incomplete, metric 404, localpref 100, valid, internal



ecc#sh ip ro 19.213.1.130

Routing entry for 19.213.1.130/32

Known via "eigrp 777", distance 90, metric 1466112, type internal

Redistributing via eigrp 777

Last update from 19.213.0.1 on GigabitEthernet2/1, 3d14h ago

Routing Descriptor Blocks:

* 19.213.0.1, from 19.213.0.1, 3d14h ago, via GigabitEthernet2/1

Route metric is 1466112, traffic share count is 1

Total delay is 55010 microseconds, minimum bandwidth is 44247 Kbit

Reliability 255/255, minimum MTU 1500 bytes

Loading 1/255, Hops 2

19.213.0.9, from 19.213.0.9, 3d14h ago, via GigabitEthernet5/1

Route metric is 1466112, traffic share count is 1

Total delay is 55010 microseconds, minimum bandwidth is 44247 Kbit

Reliability 255/255, minimum MTU 1500 bytes

Loading 1/255, Hops 2


ecc#


=============================================================================================================================



SING#

SING#

SING#sh ip ro 19.210.108.16

Routing entry for 19.210.108.16/29

Known via "ospf 301", distance 110, metric 255, type intra area

Redistributing via bgp 65000

Advertised by bgp 65000 match internal external 1 & 2 nssa-external 1 & 2 route-map DENY_DEFAULT_TO_BGP

Last update from 19.210.101.186 on Serial3/1/1, 1d17h ago

Routing Descriptor Blocks:

* 19.210.101.186, from 19.211.102.65, 1d17h ago, via Serial3/1/1

Route metric is 255, traffic share count is 1



SING#


=============================================================================================================================



JAPAN#sh ip ro 19.210.108.16

Routing entry for 19.210.108.16/29

Known via "ospf 301", distance 110, metric 404, type intra area

Redistributing via bgp 65000

Advertised by bgp 65000 match internal external 1 & 2 nssa-external 1 & 2 route-map DENY_DEFAULT_TO_BGP

Last update from 19.210.51.2 on Serial4/1/1, 1d17h ago

Routing Descriptor Blocks:

* 19.210.51.2, from 19.211.102.65, 1d17h ago, via Serial4/1/1

Route metric is 404, traffic share count is 1


JAPAN#


Giuseppe Larosa Wed, 03/11/2009 - 08:32
User Badges:
  • Super Silver, 17500 points or more
  • Hall of Fame,

    Founding Member

Hello Victor,

we are saying the same things :)


rfc 1771


d) MULTI_EXIT_DISC (Type Code 4):


This is an optional non-transitive attribute that is a four

octet non-negative integer. The value of this attribute may

be used by a BGP speaker's decision process to discriminate

among multiple exit points to a neighboring autonomous

system.


Its usage is defined in 5.1.4.


and


If received over external links, the MULTI_EXIT_DISC

attribute may be propagated over internal links to other BGP speakers

within the same AS.


in your case the routes comes from OSPF to BGP with a MED BGP attribute = OSPF metric for effect of redistribution


no problem for me, how do you call it ?

BGP metric it is fine


it can only be a BGP attribute afterall.


Hope to help

Giuseppe




lamav Wed, 03/11/2009 - 09:40
User Badges:
  • Blue, 1500 points or more

Giuseppe:


Fair enough. :-)


The reason why I make a distinction between a BGP MED and the IGP metric, as in this case as a result from OSPF redistribution, is that, when speaking of MEDs, I have always interpreted them as something that is deliberately configured to influence the path taken into an AS from another AS. It is true that the MED is reflected in the "metric" field of the BGP advertisement, but unless it was deliberately altered, I consider the case to be that MED is not being used and that the IGP metric is naturally occuring.


Lastly, just to be precise, the client is not trying to alter the IGP meteric/MED. He wants to influence the path that ECC and FMC take to get to the BGP next hop, which is SING. So, he is OK with keeping SING the BGP next hop/best path, but he wants to have ECC (if not FMC, too) route its traffic through JAPAN, in which case JAPAN will take the traffic and say "to hell with SING, I'll forward it to Australia myself, since I have a direct path."


SO, it seems that they want to use SING as the bait to attract the Australia-bound traffic, but then theyll alter the path ECC takes to get to SING by making a detour in JAPAN, and then have JAPAN forward the traffic to Australia directly on its own.


Makes sense? Is this a common approach?


Anyway, I dont want to digress.


I would still like to hear other people's suggestions for how to inlfluence the traffic to go through JAPAN.


Thanks

marikakis Wed, 03/11/2009 - 11:55
User Badges:
  • Gold, 750 points or more

I apologize in advance if I did not understand your requirements in full, but this is not a CCIE-type scenario as you stated, but one of the hard realistic scenarios an engineer has to face in real work. When words that refer to geography (Europe, USA, Australia,etc) are used in BGP scenarios, typically some heuristic technique must be used (i.e. an approximate algorithm that works in most cases and still must be periodically reevaluated).


The detour you describe (ECC->SING->JAPAN) is not going to work so easily. For traffic to have chances to move from ECC to SING, you should keep local preferences equal. But here is the BUT: If packets make it to the SING router, they will exit from the SING router. You are trying to fool BGP, but BGP is too smart sometimes :-) BGP in the SING router will prefer the locally originated routes via redistribution (best path selection algorithm step 3: Prefer the path that was locally originated via a network or aggregate BGP subcommand or through redistribution from an IGP.) An even worse thing at SING is for SING to actually prefer the OSPF learnt routes. I just mentioned the BGP logic to make clear that changing OSPF default distance won't help without additional measures.


So, you have 2 options as I see it right now:

a) Stop the BGP algorithm at SING before step 3 of BGP best path selection (assuming you forced BGP to be the protocol of choice and not OSPF using distance command). You may consider changing the weight on routes received on SING. Weight is local to SING only, and you can set a higher value for routes received from JAPAN, while setting a low value for locally originated ones. I do not like weight parameter, but if you don't want to change your local preferences in general, this is your only choice before BGP algorithm step 3.

b) Push traffic from USA directly to JAPAN (without crossing SING). I am not analyzing this further, because it seems you want the detour to occur, probably for load balancing reasons. I know at least one case we were doing such a detour with local preferences desired to be equal, but we exploited other specific properties of our topology that do not apply in your case, and it's a story longer than your own :-) I might post it somewhere some time in the future, if I have the time to write it down!

lamav Wed, 03/11/2009 - 12:04
User Badges:
  • Blue, 1500 points or more

marikakis:


Thanks for your input.


You have the scenario wrong.


The detour is not ECC to SING to JAPAN.


The traffic is already going from ECC to SING and then Australia.


The objective is to move traffic from ECC to JAPAN and Australia and get SING out of the picture.

Jon Marshall Wed, 03/11/2009 - 12:27
User Badges:
  • Super Blue, 32500 points or more
  • Hall of Fame,

    Founding Member

  • Cisco Designated VIP,

    2017 LAN, WAN

Victor


I agree with Giuseppe on this. I would use local preference on the Japan router and this will then be sent to all IBGP peers within AS65000.


I would be interested to hear what that engineer promoting EIGRP metrics has to say though. Seems to me though that if you are using BGP you may as well make use of it's capabilities to determine routing policies.


Jon

lamav Wed, 03/11/2009 - 12:46
User Badges:
  • Blue, 1500 points or more

yeah, Jon, I agree.


I mean, there are different ways to manipulate this situation. We can alter the BGP paths or manipulate the path to the existing BGP next-hop, which is what the client wants to do.


I agree with the Giuseppe, too. I would focus on changing local preference or setting the IGP metric/MED to make the JAPAN BGP peer the more favored one.



Jon Marshall Wed, 03/11/2009 - 12:52
User Badges:
  • Super Blue, 32500 points or more
  • Hall of Fame,

    Founding Member

  • Cisco Designated VIP,

    2017 LAN, WAN

I think the issue is though if you are going to go to the trouble of running BGP in your network then you may as well use BGP to influence routing policy otherwise why bother running BGP at all.


A lot of times BGP is assumed to be needed when it isn't. Now it's not clear what else is happening in that network so it may well be that BGP is an integral part of it and maybe that's why it's running. Either way, if it's there i would use it rather than messing around with IGP metric.


Jon

marikakis Wed, 03/11/2009 - 13:29
User Badges:
  • Gold, 750 points or more

Sorry if I got this scenario wrong. Still, when I read what you posted previously, I cannot undestand what you meant with the following:


SO, it seems that they want to use SING as the bait to attract the Australia-bound traffic, but then theyll alter the path ECC takes to get to SING by making a detour in JAPAN, and then have JAPAN forward the traffic to Australia directly on its own.

This pointed me to think they want to balance traffic from "anywhere" within AS6500 to JAPAN (and then Australia) over 2 paths . This can be the case if the direct link between ECC and JAPAN doesn't have enough bandwidth to handle the collective traffic to Australia. If you use local preference, a lot of traffic in the AS will use that link (e.g. from fmc). Are you sure this is what they want?


With what I suggested, traffic from fmc can go to JAPAN via SING (and then Australia), and traffic from ECC can use the direct link to JAPAN to go to Australia. This way, the direct link between ECC and JAPAN can be maintained at it's usual traffic rate.

Giuseppe Larosa Wed, 03/11/2009 - 13:39
User Badges:
  • Super Silver, 17500 points or more
  • Hall of Fame,

    Founding Member

Hello Jon,

playing with IGP metrics should always be the last resort.


Besides this, I think it doesn't work in this case without also changing the BGP attributes of the route:

the current situation of the displayed route shows that the best route has the lowest BGP metric and it is chosen even if the EIGRP metric to BGP next hop is higher then that of the other route.

This happens for the sequence of criteria used by BGP for path selection.


So changing EIGRP metric is useless in this specific case without other actions on BGP.


Hope to help

Giuseppe


marikakis Wed, 03/11/2009 - 14:01
User Badges:
  • Gold, 750 points or more

If I am wrong, then the scenario is simple enough to be handled with the local preference, as is the usual in such cases. Ask the engineer why local preference does not solve its issue and starts thinking of wild things.


Ok, the more I read this, the more I am thinking I might be right. Engineer tries to attract some traffic to JAPAN from ECC, but doesn't want all traffic in the AS to use the direct link, so he starts playing with the EIGRP metrics, instead of playing a bit with BGP on SING, because packets that reach SING have to go out SING as I said before. A little work on SING can stop traffic from exiting via SING, as I described previously (SING2 might also need some work to avoid a loop).

Jon Marshall Wed, 03/11/2009 - 14:39
User Badges:
  • Super Blue, 32500 points or more
  • Hall of Fame,

    Founding Member

  • Cisco Designated VIP,

    2017 LAN, WAN

Giuseppe


"playing with IGP metrics should always be the last resort"


I totally agree. Was there something i wrote that suggested otherwise because i thought i was suggesting the same as you ie. use local preference.


Jon

marikakis Wed, 03/11/2009 - 15:07
User Badges:
  • Gold, 750 points or more

Most people don't play with metrics for fun because they lead to confusing and unscalable setups. It looks like this guy is in a last resort situation (load balancing internally issues are not uncommon) and might actually make it in the end. All this guy needs for load balancing to work is to reset the MEDs to zero. Then best path selection algorithm will go further down, and the lowest IGP metric to the BGP next hop from each router in the AS will be used to balance the traffic, so is close to a solution that will distribute traffic over the links to Australia. When someone suggests the weird, try not think they don't know what they are doing.

lamav Wed, 03/11/2009 - 17:06
User Badges:
  • Blue, 1500 points or more

Giuseppe:


"Besides this, I think it doesn't work in this case without also changing the BGP attributes of the route:

the current situation of the displayed route shows that the best route has the lowest BGP metric and it is chosen even if the EIGRP metric to BGP next hop is higher then that of the other route."


I think you are confusing the issue. The client is not trying to influence the BGP selection process. The client is satisfied with leaving the SING router as the BGP next-hop for ECC and FMC to reach Australia. What the client is trying to change is the eigrp route from the FMC and ECC routers to the SING router, and that has nothing to do with the Australia IGP metrics that the SING router is advertising.

Jon Marshall Wed, 03/11/2009 - 18:13
User Badges:
  • Super Blue, 32500 points or more
  • Hall of Fame,

    Founding Member

  • Cisco Designated VIP,

    2017 LAN, WAN

I'm confused now :-).


My understanding was that FMC & ECC have IBGP routes to the Australian networks which have been redistributed from OSPF.


So what i think Giuseppe is saying is that it doesn't matter what you do to the EIGRP metrics, FMC & ECC will choose the best path from the IBGP routes. The best path is SING and that is the one they will use.


Now i thought the question was how to make FMC & ECC see JAPAN as the best route and the answer is local pref. This is by far the cleanest solution to the problem and i would have thought the most logical for anybody else coming along later to understand.


Having FMC & ECC see SING as the best route but sending the traffic via JAPAN so that JAPAN sends the traffic is just overcomplicating things. If you were the next network administrator coming along that would not be an obvious solution to the initial problem.


The other point i was making is if the client wants to mess around with metrics why bother with BGP at all. Just run OSPF on all routers and then influence the OSPF metrics directly.


Maybe i'm missing something here and as Maria pointed out it's not that i'm assuming the engineer proposing the EIGRP solution doesn't know what he is doing. He may have a very good reason for what he is proposing. It's just not obvious at the moment.


Jon

lamav Wed, 03/11/2009 - 19:08
User Badges:
  • Blue, 1500 points or more

"My understanding was that FMC & ECC have IBGP routes to the Australian networks which have been redistributed from OSPF."


Correct.


"So what i think Giuseppe is saying is that it doesn't matter what you do to the EIGRP metrics, FMC & ECC will choose the best path from the IBGP routes. The best path is SING and that is the one they will use."


Correct, although I think Giuseppe was saying something else. I think he is under the impression that the purpose the client has of changing the eigrp delay metric on the ECC and JAPAN routers is to affect the selection of the BGP next-hop, which of course is completely wrong. They are not trying to do that at all. The IGP metric they want to change is the EIGRP delay so that ECC and FMC take a route through JAPAN on their way to SING, which of course they never reach because JAPAN is going to send the traffic directly over to Australia itself. If that's not what he means, then he is correct, too. Sorry, Giuseppe, your foreign English throws me off sometimes. :-)



"Now i thought the question was how to make FMC & ECC see JAPAN as the best route and the answer is local pref. This is by far the cleanest solution to the problem and i would have thought the most logical for anybody else coming along later to understand."


Correct, once again. And I agree 100%.



"Having FMC & ECC see SING as the best route but sending the traffic via JAPAN so that JAPAN sends the traffic is just overcomplicating things. If you were the next network administrator coming along that would not be an obvious solution to the initial problem."



Correct and I agree. I would not take this approach. I would take the more direct approach of making JAPAN the next-hop router by manilpulating the BGP attributes (local pref, etc).



"The other point i was making is if the client wants to mess around with metrics why bother with BGP at all. Just run OSPF on all routers and then influence the OSPF metrics directly."


I agree and understood your point.



"Maybe i'm missing something here and as Maria pointed out it's not that i'm assuming the engineer proposing the EIGRP solution doesn't know what he is doing. He may have a very good reason for what he is proposing. It's just not obvious at the moment."


You are not missing anything, given what I have told you, which is all I know atthis moment.


You are not confused but right on target.


Victor

marikakis Wed, 03/11/2009 - 22:41
User Badges:
  • Gold, 750 points or more

Victor,


I am not the type of person that insists on something just so my own solution is the selected one (and so I get the points in this case)! I am more practical than this, and I only care if an issue is resolved, no matter who does it! I was actually very busy last night, but still decided to respond, because I thought I might help.


You have not still responded in my previous post about what using SING as "bait" and "detour" mean. If those were the words of the client, then the client might be trying to tell you something here. This sounds to me as if some traffic to Australia is the 'mouse', SING is the 'cheese' and detour is the path via SING2 to JAPAN. The mouse goes to the cheese, and instead of trapping and killing it, you reroute the mouse to go out of the house (AS) through the JAPAN door to live in Australia! This metaphor is so cute, I think I am going to cry in the end if I am wrong :-)


This issue is either very simple and can be solved with local preference us others have suggested from the beginning or is not.


If it is not, I have already provided guidelines. It is common for one type of traffic change to affect other links. This network was using the SING path to go to Australia. When they switched the traffic to go via JAPAN (probably with local preference), they might have noticed the direct link between ECC and JAPAN got overloaded. It is very hard doing traffic engineering using local preference, because it causes a lot of traffic to go in one direction only. If IGP metric to BGP next-hop is used (after reseting the MEDs to zero or make them equal), you could have some type of closest exit routing (and you can overcome altogether the exit via SING, by turning traffic to go via SING2 to JAPAN).


Kind Regards,

M.

marikakis Wed, 03/11/2009 - 23:31
User Badges:
  • Gold, 750 points or more

Note also, that setting local preference will cause traffic to Australia through SING router (which may have other sources of traffic for Australia via other of its interfaces), will choose to go to Australia via JAPAN, and will not use the direct link it has with Australia (which might not be a desired thing to happen). Unless OSPF steps in and fixes this.


Jon, maybe my phrase "don't know what they are doing" sounded somewhat rude, but written speech has those type of issues. I know how helpful all of you are. I meant that when someone tries to do weird things, those things might point to the problem at hand. Sometimes clients provide little information or if they are engineers assume you will understand what they are getting at by using jargon. And such issues are a little bit of a chess backtracking procedure: if this is their move now, what was the situation in the chessboard before they made this last move?

Also, when I said previously that client does "wild things", in the same sentence I said to ask why local preference does not resolve its issue, so there I assume client probably already did that and saw something bad happening, so tries to seek consultancy, because from this point the case is not trivial.

marikakis Thu, 03/12/2009 - 01:40
User Badges:
  • Gold, 750 points or more

One last probability is the following: There is not a problem with the bandwidth utilization of any links in the AS after setting local preference, but some traffic that reaches SING via some of its interfaces (e.g. from SING2 to SING in the case SING2 does not have full routing, I am not even sure if SING2 runs BGP at all from the diagram) keeps exiting via SING. OSPF might be the reason for this and changing the administrative distance for OSPF or BGP on SING might solve this. This is a more normal scenario than the one I was thinking up to now. It also explains a bit why the client is trying to attract traffic towards JAPAN by changing EIGRP metrics. Some type of traffic might keep exiting via SING even after setting local preference.

marikakis Thu, 03/12/2009 - 02:06
User Badges:
  • Gold, 750 points or more

If this is the case, then the mouse story changes a bit and becomes: mouse comes from SING2 (SING2 might just have some default towards SING) to cheese (i.e. SING) and needs to be diverted towards FMC and run a circle to exit via the JAPAN door.

lamav Thu, 03/12/2009 - 06:22
User Badges:
  • Blue, 1500 points or more

Mari:


Easy there, young lady! I appreciate your efforts. I didnt respond to you because I was in the middle of an in-depth discussion with Giuseppe and Jon, which can actually help you understand exactly whats going on if you read it.


Im not even going to attempt to unravel your mouse, cheese and house analogy. Sorry, but it is way too complicated for a simpleton, like me. :-)


I think we may have beat this topic to death by now. I will talk to the client to try to unravel his logic and see if he is making any sense.


Thanks, folks!


Victor



marikakis Thu, 03/12/2009 - 13:57
User Badges:
  • Gold, 750 points or more

Victor,


I am almost 32 now and not sure if this is very young! :-) I have read the posts again and I won't mention the load balancing scenario again, although it is not an unrealistic requirement for the future if the JAPAN link to Australia starts experiencing similar issues as the SING link to Australia. In such a case, resetting the MEDs and using closest exit routing using IGP metrics might be a solution, so keep that in mind for future reference. In the ISP environment I was working in the past such issues were very common, so be prepared because this network is not very much different in logic as potential segment of an ISP network.


Typically, an engineer who has BGP configured in various countries and in more than say 2 routers, has to know BGP better than an engineer who only has 1-2 BGP connections in a single router towards its upstream(s). Local preference is the first thing an engineer that works with BGP learns to manipulate traffic. Such an engineer might get frustrated if the consultant that is giving them advice suggests the typical solution, because it might have been the very first thing that crossed their minds as well and they called for help to hear something more.


From my experience, there are particular phrases in client information that are key to see the solution, even if non-technical terms are used, so a consultant has to notice all the details to avoid frustrating the client and be prepared for various potential issues and solutions to give to the client directly or be prepared to ask the client the right questions. In this case I might went too far with my imagination, but you asked for more opinions in the first place, and I wanted to inform about potential issues. Clients sometimes might know their networks so well, that they don't bother to mention important details, because in their heads they are taking them for granted, and speak as if you knew their networks from day one and you can figure what they need now without expicit stating. I might have overestimated the client's knowledge in this case, but I have seen underestimation of others leading to worse results (i.e. we would not call our consultants anymore, because we knew they would suggest something we did 10 steps before).


In the end, it is always good to hear that problems are less hard than expected and typical solutions can be used. There is a well know saying that suggests to keep things simple, and I try to follow that. Whatever I might imagine, I don't put to practice for no good reason.


Kind Regards,

Maria

marikakis Thu, 03/12/2009 - 15:02
User Badges:
  • Gold, 750 points or more

If you look back in this conversation, you will see all I was trying to do was to through some ideas before client responds, so you are more prepared and client keeps calling you in the future. I could just wait for the client to respond and wash the dishes instead like a good lady! :-)

marikakis Fri, 03/13/2009 - 12:01
User Badges:
  • Gold, 750 points or more

Victor,


In another case you opened, I see that you are now making the MEDs equal. It would be nice to hear what happened in this case and why for future reference when you are done with this and you have more time, not only for all of us, but also because a conversation (although admittedly long) that has the specific title you chose in the first place is expected to be considered interesting for many people. :-)


Kind Regards,

Maria

rpfinneran Sat, 03/14/2009 - 22:43
User Badges:
  • Bronze, 100 points or more

I would simply set the local preference at the Japan and Sing routers during the redistribution, since they are in the same AS as the fmc and ecc routers. You could do this using a route-map to ensure it only affects traffic destined for Australia, and no other connections that my not be shown in your drawing. Good luck

marikakis Sun, 03/15/2009 - 04:09
User Badges:
  • Gold, 750 points or more

As I have said before, local preference causes a lot of traffic to move in one direction only. The SIG link to Australia has utilization issues. If someone sets the local preference to move a lot of traffic towards the JAPAN link to Australia, in the best case they are effectively moving to the near future the utilization issue from the one link to the other. There are utilization issues in this network and the question is not just how to route the traffic, but also to consider all the link utilizations. This is the vital information that has not been provided by the client from the beginning.


If they are now choosing closest exit routing by restting the MEDs, they are choosing a more long term solution that can be fine tuned. What would they do if the other link starts experiencing the same issues in the future (if not already experiencing or some other link does)?


Also, some large networks have well defined policies and handle higher level routing decisions with local preference (such as choice between client, peer, upstream routes), and would not be willing to have different local preferences here and there when there exist other ways to do load balancing.


I guess local preference is elegant when you have a couple of routers or link utilizations are not considered, as is usually the case with exams. Although this looks like an exam question from the title, it actually refers to a real network with link utilization being an issue. Another cool thing with exams is that no angry customers are calling while you are trying to set up the network and you have hours to save the world, while in a real network you might have only a couple of minutes or seconds. In those minutes, it is easier to think about the bandwidths and the utilizations of links, than trying to remember what local preference you have set for each route, where in the network have you set it and why (an approach clearly unmanagable).

marikakis Sun, 03/15/2009 - 05:03
User Badges:
  • Gold, 750 points or more

Guiseppe,


I would really like to read your opinion on this last point, if you have the time, since you deal with big customers and you typically have something good to say in such issues. I feel like a lone rider in here, although I am thinking that if I was really wrong, you would have stepped in.


Kind Regards,

Maria

Jon Marshall Sun, 03/15/2009 - 12:27
User Badges:
  • Super Blue, 32500 points or more
  • Hall of Fame,

    Founding Member

  • Cisco Designated VIP,

    2017 LAN, WAN

Maria


No Giuseppe but hope you don't mind a few comments. I have always found your posts to be very helpful and full of technical detail. And i also consider you to have more knowledge than i do with BGP. In addition as far as i am aware nobody is saying you are wrong, as often with these sort of things right and wrong don't come into it.


But the original problem was how to send all traffic via JAPAN, that was the customer requirement. It was not how to load-balance between the 2 links. If it was then i don't think any of us would have suggested local preference as the simplest solution.


With the people involved in this thread i have never seen them assume that people do not know what they are talking about and i would hope i haven't either. So to say we are underestimating the engineers knowledge is wrong. Perhaps this engineer has no or limited experience with BGP and is a lot more comfortable with EIGRP. Perhaps that's why Victor is being asked to look at the issue as well.

Then again, perhaps the engineer sees something we don't, perhaps Victor has been given incomplete requirements, perhaps as you say the engineer has taken for granted a lot of things about the network that he has not passed on. But they are all perhaps.


When you say "I am not the type of person that insists on something just so my own solution is the selected one (and so I get the points in this case)!"


that implies that there are people who do this. Again i would say as far as this post is concerned i am not aware of that happening although perhaps you feel differently. I agree 100% with you as i'm absolutely sure the others do ie. the important thing is to get the right answer.


We all use our experience as well as knowledge to try to help people but we all have different experiences.


Jon




marikakis Sun, 03/15/2009 - 12:53
User Badges:
  • Gold, 750 points or more

Jon,


I know it might look like I am assuming a lot in this case, but sometimes you can call this instict. Instict is typically associated with women, but is essentially a quick analysis of various factors that you cannot phrase at the time your brain processes all the available information. I had a feeling there was some load balancing requirement in this, as is typical in cases of 2 links with one overflowing. Also, Victor said at the very beginning that he is not familiar with the network yet or what the client is trying to, asked what people think although he already got answers, and then said I was wrong. I still cannot understand this. How can someone be sure I was wrong at this point? Unless the idea was me to confirm that he was right (although he doubted that himself) and I didn't catch it.


I won't insist on anything, although when many solutions exist, one can be more managable and scalable than the other. This is not my network after all and if anyone insists on anything they should be ready to accept the consequences of what they are doing. Sometimes people in the forum say things in an academic fashion (I know I have done that mistake as well), only because they feel more comfortable when their own network is not discussed. Still, I sometimes feel like the network is really mine and try to think of what could go wrong. And yes, people can insist just to get the points. I say what I think and I don't care about that, so I don't like the other person to think that of me and ignore me when I am only trying to help.


Thank you for your reply. I don't feel a lone rider anymore and I appreciate you answering because Guiseppe might not catch the thread after becoming so long. I do appreciate your answer. I asked Guiseppe because we typically meet in such cases and he has no problem telling me I am wrong in something :-)


Kind Regards,

Maria


p.s. To add to the instict: it is typical for people to be confused with the difference between metrics and MED even if they know BGP well enough, so I thought this guy was only missing just one detail to do is job.


p.s.2 I just rated your post, although I usually don't say this when I do it for various reasons. I just noticed that you thought I was referring to someone in this thread about the points, which is clearly a misunderstanding. Jon, I have to say you are underestimating me now :-) I know what you guys are doing in this forum!

Giuseppe Larosa Sun, 03/15/2009 - 13:33
User Badges:
  • Super Silver, 17500 points or more
  • Hall of Fame,

    Founding Member

Hello Maria, Jon,


I have been out all the afternoon.


As Jon has written no one here thinks to be better then others, or assumes no knowledge of the subject on the other people.


I can say that Maria's posts are usually high quality posts with in depth analysis.

These posts make me think I should write less posts of better quality.


I think Victor had few details about network and customer needs, and he was under pressure to provide a feedback.


He was puzzled for the customer engineer proposal of changing EIGRP metric because he feels it cannot influence current best BGP path choice, at least without other changes in BGP attributes of involved routes.


And everyone here agrees on this.

I suggested to use local-preference, Victor asked for other opinions and he has received Jon's answer and your posts.


I think that Victor has just tried to say that he had no time to follow your suggestions (as he had no tim:e to discuss how OSPF metrics are carried inside BGP :) ... )


Be aware that this doesn't mean you (Maria) are wrong in your "instinct": what you suggest can be the real customer needs/objectives or it can be the next customer needs/objectives if the Japan link becomes overwhelmed.


As a last note:

local-preference can be modified on a per prefix basis so some prefix based administrative load sharing can be implemented in order to achieve the desired level of usage of the two exit points.


if the BGP MED are made equal and the IGP metrics in the EIGRP domain are used to decide how to use the exit points there is no added value in having implemented an iBGP full mesh: it is like having redistributed OSPF routes directly in the EIGRP domain.


Again Jon had noted in one of his first posts that we don't know why the customer has implemented BGP.


Once that iBGP has been deployed and it is in use it is common practice to use its rich set of tools to influence routing paths.


Hope to help

Giuseppe


marikakis Sun, 03/15/2009 - 13:42
User Badges:
  • Gold, 750 points or more

I can only say I really love you guys! :-)


p.s. I have to work on my thesis at this point, so I try not to start conversations I am not ready to complete. That's why I post less :-)

lamav Sun, 03/15/2009 - 16:57
User Badges:
  • Blue, 1500 points or more

Folks:


This is no place to be thin-skinned. There are a lot of ideas, comments and suggestions going back and forth and a lot of good things being "said" by a lot of people. Therefore, its not always eays to entertain everyones suggestion or comment extensively on everyones opinion. Someone asks for opinions, you give it, and they either acknowledge it or not; that's it. No need for long, drawn out explanations about men, women, etc etc etc....its all inconsequential.


As for the network, yes, my knowledge of this particular segment of the network --- especially at the time I introduced this topic -- wasnt too great. I had never seen any of these routers or this part of the clients network before. I made that perfectly clear in my initial post, or shortlay afterward.


Moreover, the client did not specifically ask for MY feedback. But, I learned that the client wanted to achieve a certain affect and that he was going to proceed in what seemed like a sub-optimal manner, so as a consulatnt I took it upon myself to offer some advice. I also made it perfectly clear that I had not had a chance to ask the client exactly what he was thinking and why he wants to proceed that way. I did have my suspicion, though, that the plan was not even hatched by an engineer, but by a project manager who was being adventurous and perhaps oversimplifying things. So, I proceeded pro-actively to see what the plan was and I also gave my opinions on it. No biggie.


I posted my scenario on this board because I know there are some pretty damn smart people on here with a lot of experience: Jon, Giuseppe, Rick Burts, Edison, Harold Ritter, Pablo, Joseph Doherty, and many more, so I wanted to hear a variety of opinions, including Maria's, who until this thread, I actually had never seen before. But Im glad she gave her opinion.



marikakis Sun, 03/15/2009 - 23:30
User Badges:
  • Gold, 750 points or more

Victor,


I do consider my pure technical posts better than the others, so I knew many of my posts here were destined to not be liked even by me :-)


I guess I caught some fire when you called me a young lady! You probably meant well, but as we have seen before in this case, non-technical comments can cause misunderstandings.


Thank you for opening this case, as I always have fun with such cases no matter what.


Kind Regards,

Maria

lamav Tue, 03/17/2009 - 09:37
User Badges:
  • Blue, 1500 points or more

OK, here goes.


I just got off the phone with the client and here is the scoop.


What they want is to deliberately install some asymmetric routing into the mix.


Traffic from Australia and destined for ECC and FMC (U.S.) will always take the route through SING because it is advertising a default route and there are no specific routes in the Australia routing tables for the US. SING and JAPAN are ASBRs and they both inject default routes into Australia, but the better metric is through SING.


On the other hand, they want to manipulate the delay to force traffic destined for Australia from the US to go through JAPAN, as we discussed all along (now we're talking about the other direction).


The changed the delay on the ECC and JAPAN routers, and now some of them Australia-bound traffic goes through JAPAN, but most of it still goes through SING. So what they want to do is further this solution by maniipulating the seed metric in OSPF in Australia BEFORE its redistributed into BGP so that BGP can pass the metric they want to the US through its BGP neighborships.


Personally, I think this is a convoluted approach. I would not deliberately install asymmetric routing and bother manipulating the metrics in such a seemingly random fashion to make it work. I would manipulate BGP atttributes, as we discussed before.


What do you guys think of their solution?

Giuseppe Larosa Tue, 03/17/2009 - 11:16
User Badges:
  • Super Silver, 17500 points or more
  • Hall of Fame,

    Founding Member

Hello Victor,


I agree I would use BGP attributes they are acting as BGP wasn't present.

Unless they are also planning to remove BGP this approach requires more work and provides less control.


With BGP you can easily decide on a per prefix basis what would be the preferred exit point.


Playing with IGP metrics has a greater impact: when it works it can move all traffic to that exit point.


I'm not surprised that most traffic is still going via Sing router ...


Hope to help

Giuseppe


Mohamed Sobair Tue, 03/17/2009 - 10:47
User Badges:
  • Gold, 750 points or more

Hi Vector , ALL,


There are multiple approaches to achieve thier desired goal.


I would personally choose to modify the Local preference in Japan router to influence outgoing path On the IBGP dommain.


Still modifying Eigrp delay at SING router facing FMC, I can see that they are 2 links used by Eigrp , Modifying the delay would also reflect the total metric recieved by ECC and Japan, but would require additional config if you want to achieve loadbalancing the traffic across both serial interfaces, In other word you will have to use the "variance" command, Increasing the Delay at SING router is another approach, ECC and Japan router would then recieve the path to network 19.210.x.x with higher metric than japan does, thus prefering japan Route, but the easiest way would be modifying the local preference at Japan router.


As Maria noted earlier, what if they want in future to chang the path or if the path gets overwhelmed? IS there any sort of loadsharing here? What are the exact Network redistributed by ACR03 and ACR01? How (SING and SING2), (SING2 and Japan) connected?



HTH

Mohamed

lamav Tue, 03/17/2009 - 10:51
User Badges:
  • Blue, 1500 points or more

Mohammed:


Long time...


Well, they are viewing this as a temporary solution.


They want traffic bound for Australia to go through JAPAN and traffic in the other direction to go through SING.


So they reduce the delay on ECC and JAPAN to force ECC and FMC to go through JAPAN to get to Australia.


So their idea of load balancing/sharing is to have traffic use one route in one direction, and a separate route in the other.

Actions

This Discussion