Network schema

[suthomas1]

Hi,

Attached is one of our remote sites schema.

R1,R2 are provider termination routers at loc B. Router A is similarly device at loc A.MPLS cloud is the connecting link.

Both R1,R2 are terminated locally at loc B on a Layer3(6500) switch.

Now before its discussed, it would be better for me to let all know that this scenario(but not in current form, was initial stages for ideas) was put up earlier here & had very good feedbacks from Guisseppe, Joseph et al.Situation is this being remote site, i wasnt really involved in implementing this & due to problems now, it has been put back to me for further checks.The objectives for this site were:

-achieve inbound & outbound loadbalancing on both links alongwith redundancy.

Current setup:

R1,R2 are enabled with glbp for outbound loadsharing.Inbound is not really in our control as provider is involved between.They say inbound loadbalancing and/or symmetric traffic cant really be achieved as bgp uses best path always.

So they had to start with existing setup to meet their site deadline.Location B routers have gre tunnel pointing to Loc A router.

Problems:

Reports say there is no real loadsharing happening inbound, most of the time it uses the 2nd link which also is the glbp active one.Outbound is reportedly more again with glbp active(2nd link), but very small bouts can be seen on first link as well.Please help me with your views.

1.How can i ascertain if outbound using glbp is indeed loadsharing on both links,as i understand that since the L3 switch is the one giving out arp & is seen as one & new host arp request would only come to L3 ports hooked to internal network; R1,R2 would rarely get newer arp replies to change link path(Correct if this is wrong).Any commands or other things to check with?

2.How to see if inbound loadsharing is happening , any commands or other things?

3.Would the failover time be glbp hold timer in lan side & bgp hold time in wan side(3 mins)?

4.Lastly, any suggestions , configs et al to actually change this to achieve the desired requirements of inbound & outbound loadbalancing alongwith symmetric flow if possible.

Thanks in advance.

[Joseph W. Doherty]

Could you clarify routing between locations A and B? You mention both BGP and GRE tunnels?

On point 4, you're generally not going to be able to guarantee symmetric traffic flows and load balancing. This usually isn't an issue across WANs. What you do often want to avoid is per-packet load balancing.

On point 1, recall there's some (new?) GLBP command that will show you relationship between hosts and GLBP virtual gateways. I doubt you need to verify this since GLBP generally works, but since it directs all host traffic, one host that sends much traffic could skew stats, especially within shorter time frames. If this is an issue, routing load balancing might work better especially with CEF since it will balance flows.

On point 3, GLBP failover should be within seconds using defaults. Also defaults can be changed. (NB: Don't recall whether subsecond failover is supported with GLBP.)

For BGP, default failover can be "slow" especially in some situations. Failover time also depends on nature of failure, e.g. lost peer vs. lost link. BGP timers can also be changed, but BGP convergence generally can't be set as fast as most IGPs support.

On point 2, inbound traffic stats should reveal load balancing. Again, except for per-packet load balancing, individual heavy flows can skew stats, again, especially during short time periods. Across a day or more, load balancing should tend toward expected proportions.

[suthomas1]

BGP is used as normally for provider to interact with their PE cloud. GRE is used to move all default routes across the other site, as most of the resources including internet for site Bis accessed via Loc A.

- By avoiding per-packet, u mean we should use per session correct?How can we do that in correct case?

-But wouldnt single L3 switch cause issue with arp for glbp?

-yes..glbp does support failure in milliseconds.

Any other way we can change the design/configurations to be more efficient with existing configuration or totally new one

Thanks.

[Joseph W. Doherty]

Sorry, still somewhat confused with BGP and tunnels. What I'm trying to understand, are locations A and B only "known" to each other via routing across tunnels or are they also "known" via routing across MPLS cloud. If the former, BGP should be a non-issue except to move GRE packets between locations A and B, and BGP load balancing should be also be a non-issue.

"By avoiding per-packet, u mean we should use per session correct?How can we do that in correct case?"

CEF offers per-packet as an option. If you don't enable it, all packets within the same flow will take the same path.

"But wouldnt single L3 switch cause issue with arp for glbp? "

Depends whether the L3 switch is acting as a L2 switch or router with regard to host traffic. If the former, not an issue. If the latter, GLBP shouldn't be used; either use dynamic routing, or mHSRP with multiple static routes.

[suthomas1]

They are known via routing across mpls via bgp as well.Sorry, if am still not answering your question!

CEF is enabled on the l3 switch..so it should be per-session correct?

L3 is further connected to 3750 switches inside the network to which hosts hookup.

Thanks.

[Giuseppe Larosa]

Hello Sunny,

>> L3 is further connected to 3750 switches inside the network to which hosts hookup

so GLBP load balancing is defeated once L3 switch makes an ARP request it gets a single AVF MAC address in response and use only one router as exit point.

As Joseph has suggested or you move to a dynamic routing protocol between L3 switch and R1 and R2 or you use multiple default static routes with next-hops = HSRP VIPs of two different groups.

It is still no clear what is the usage of the tunnels are they going via the internet and should act as backup paths?

About load balancing from service provider to R1 and R2: this is technically possible if using

eibgp maximum-paths on remote PE node (the one connecting to HQ) in address-family ipv4 vrf your.customer

To achieve this R1 and R2 has to be connected to two different PE nodes and a different RD is used.

So both BGP advertisements can travel in service provider signalling infrastructure.

So you may try to ask again to them to do so.

About per packet load-balancing there are known drawbacks especially with VOIP traffic.

Hope to help

Giuseppe

[suthomas1]

Below one is for clearing any confusion that i may be causing to you all..

L3 switch has ports as access vlan ports & SVI for these vlan's & 3750 where users connect physically has ports assigned to certain of these vlans.

So the L3 here acts as gateway for all hosts,correct?

1.If so, as Giuseppe said about arp request by L3, i would appreciate if a bit more light is thrown as to how the arp works in this case & how technically it defeats the purpose of glbp, involving 2 routers, L3 & 3750 , hosts.

2.On tunnel, i missed out one info earlier, loc A has two tunnels to both links at loc B.

Tunnels don't go over internet, it just pushes default route across to loc A.

If am still not being clear on tunnel /bgp, i can give output for specific required commands.

3.about ebgp multipath on PE.. as told by provider, 2 paths are seen from PE in loc A using both ckts in Loc B.

so does that mean loadsharing inbound with bgp is ok now?

Following is the brief output that they gave us.

10.72.0.0/16 & 10.72.1.0/27 are subnets off Loc A ;10.84.0.0/16 & 10.84.1.0/27 are local subnets at Loc B.

>>>>>

sho ip bgp vpnv4 vrf V105

Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,

r RIB-failure, S Stale

Origin codes: i - IGP, e - EGP, ? - incomplete

Network Next Hop Metric LocPrf Weight Path

Route Distinguisher: 65000:105 (default for vrf V105)

*>i0.0.0.0 165.14.75.4 0 100 0 1 i

*> 10.72.0.0/16 28.16.74.10 0 0 1 ?

*> 10.72.1.0/27 28.16.74.10 0 0 1 ?

* i10.84.0.0/16 67.10.44.1 0 100 0 6500 ?

*>i 165.14.76.7 0 100 0 6500 ?

* i10.84.1.0/27 67.10.44.1 0 100 0 6500 ?

*>i 165.14.76.7 0 100 0 6500 ?

Thanks.

[Giuseppe Larosa]

Hello Sunny,

1)

The question with GLBP load balancing is that it works in the following manner:

the AVG (Active Virtual Gateway) answers to ARP requests for the VIP address (the default gateway) using some form of load balancing among multiple AVFs (Active Virtual Forwarders): it provides in different ARP replies answers different AVF MAC addresses. For example it can use the list of AVF MAC addresses in round robin manner.

AVF1 in answer N

AVF2 in answer N+1

AVF1 in answer N+2

and so on

So GLBP load balancing work well when there are multiple clients in the VLAN.

GBLP betweeen L3 switch and R1 and R2 provides redundancy but not load balancing if L3 switch is the only host for R1 and R2.

Just to say used on L3 switches towards clients is fine.

2)

so you use a GRE tunnel for traffic directed towards the internet (default route) ?

3)

to be sure remote PE performs load balancing the provider should look at the sh ip route vrf V105

the sh ip bgp vpvn4 vrf V105 tells that the two advertisements reach the remote PE connected to HQ (this means different RDs have been used ) but it is not enough to tell if both are used.

To be clear the path with > is best.

We can say there is a potential for load balancing but only sh ip route 10.84.0.0 can tell us if both paths are installed (this depends on the command eibgp maximum-paths 2 given on remote PE af vrf V105)

Hope to help

Giuseppe

[suthomas1]

Hi Giuseppe,

1. Means,as L3 is single handedly talking with R1R2 on behalf of hosts ( but multiple requests from hosts for arp would be handled by L3 only), L3 would either ask for arp when the existing entry timesout or new flow is found.

If neither happens,R1R2 just sees L3 ip/arp as single entry and since the L3 arp wouldnt possibly timeout because of stream of data, there wont be real balancing.Correct?

Just to add on, the usage upstream on both links if checked, shows link 2(glbp active) being used more.

2.yes all unknown traffic is moved across tunnel towards the loc A which then routes it over internet.

3.both these outputs we would have to check on provider PE right?

Thanks.

[Giuseppe Larosa]

Hello Sunny,

1) yes the point is that a router/multilayer switch makes use of ARP to resolve an ip next-hop of a static default route 0.0.0.0/0.

So if L3 switch has a default route towards GLBP-VIP it makes an ARP request and stores the reply for 4 hours.

2) now it is more clear the usage of tunnels.

3) yes on PE node he/she should send you the output of

sh ip route vrf V105

and

sh ip route vrf V105 10.84.0.0

to confirm effective load balancing

Hope to help

Giuseppe

[suthomas1]

1.Would arp & cam table age have any difference wrt l3/routers?

considering this behaviour, any move of traffic across other link would only be fine if either the arp timesout early.& if it does hold on for its entire age, only then may it move to other link?

Thanks.

[Joseph W. Doherty]

"They are known via routing across mpls via bgp as well."

If the internal LAN addresses are known by both MPLS/BGP and via the GRE tunnels, then it's unclear which the router uses.

"CEF is enabled on the l3 switch..so it should be per-session correct? "

That should be the default.

"L3 is further connected to 3750 switches inside the network to which hosts hookup. "

If your hosts are transiting a L3 hop before than reach the GLBP gateway, i.e. a router's interface is the "host" to GLBP, they won't effectively load balance.

[suthomas1]

Joseph/Guisseppe,

Thanks for the valuable inputs.Same schema, i tried some verifications on this.

When one of the interface on L3 is shutoff, this makes existing R2(glbp active) to change state & has R1 as active.

In my opinion outgoing traffic from clients would continue fine thru the recent active one.

But my concern/doubt is:

1.On return path. inbound traffic wouldnt come back to clients, as when it reaches the router, router(glbp) would see no arp from the switch path which was previously glbp active..as its shut & as the arp for previous assigned active forwarder would still be there on L3.This may cause the traffic to stay still.

is this assumption correct?

2.Another question is since its unsure which router inbound will the traffic use to come back to clients, this may or may not cause a problem always.If return uses same router as it was before glbp active switchover it may cause problems as it misses arp. But if it uses other router, it may get through?

3.Any way we can check to see which router path the return traffic takes after turning of one interface on switch?

Please correct if this is wrong with the way it would behave.

Thanks.

[Giuseppe Larosa]

Hello Sunny,

the problem is not so dramatic a router/multilayer switch is allowed to ARP for a host not present in the ARP table so asymmetric routing is normally supported.

the first packet for a host missing in ARP table is sent to main cpu for ARP processing ("CEF punted") subsequent packets of the same flow are hold in some buffers once the ARP entry is built the packets can be forwarded.

It shouldn't be a problem with TCP traffic during TCP session setup there is also the time to create the missing ARP entry.

An UDP video streaming flow can experience some initial drops but again it should be acceptable.

Bearer channels of a VOIP communication using RTP/UDP can suffer this initial ARP activity but they are relatively slow.

Hope to help

Giuseppe

[Joseph W. Doherty]

The only issue that might arise with asymetrically routing, if there were different L2 switches involved and you then had unicast flooding. GLBP should be an non-issue for the returning traffic.