BGP asymmetric routing problem

[sstone]

I am having an issue with asymmetric routing that I cannot get a handle on. I have an Internet VLAN with a PIX 525 and two Cisco 3825s. One 3825 connects to AT&T and one connects to Sprint, running eBGP externally on both and iBGP in between. Default gateway of PIX is HSRP gateway (primarily active on my AT&T router). I receive partial+default routes from both providers, but routing to certain remote networks is not working. [Note: I'm also setting routes received from AT&T with a local pref of 150]

The scenario seems to result when an asymmetric routing path is created, where a route learned from Sprint is used for an outbound destination host and that remote host tries to send its return traffic via my AT&T connection. I've been logging traffic received from this remote host and occasionally will see the return traffic permitted by my AT&T router, but not always. The end user behind the PIX does not see anything. When I force all traffic through either AT&T or Sprint it works fine.

Just some more info, the network in question is a class C network such as 63.x.y.0/24. The route from Sprint that is being preferred by my routers is a broad route - 63.0.0.0/9. I don't understand the implications of aggregation too well - could this be part of the problem? Would accepting full routes from one or both providers be advantagous?

Any insights would be appreciated.

Thanks,

Seth

[tdrais]

I guess the first question is do you own the /24 block or is it owned by one of the ISP's. And second if it is owned by the ISP do they allow you to advertise it out another ISP.

If we assume you are allowed to use it as you please you have a couple of issues.

First local preference only affects outbound traffic. You need to prepend ASPATH as you send the routes to affect inbound traffic from multiple AS.

Next you actually will need 2 /24 block if you want some traffic to go one way and some to go the other. The smallest mask most ISP's allow is /24 so although you could break your /24 into 2 /25 and advertise them out most ISP's will drop them as they get them or their upstream neighbors will drop them.

You final problem will be how you make sprint ever win over ATT if you send out the /24. You will need to get sprint to advertise out the /24 as well as their summary. If both are equal you have a chance to influence routing using ASPATH but if you do manage to get att to advertise the /24 they will always win because of the longest prefix first rule.

Having a summary like this makes it a pain. You sorta get random results. It depends where on the internet you are as to what you are seeing. If you were to go inside of sprint or one of the smaller ISP's that use sprint as their primary I suspect you will see and use the more specfic routes. Outside you will only see the summary.

[sstone]

So if I have just a single /24, is there any possibility for load sharing (outbound or inbound)? I was told that I could for outbound, but not inbound and that I would have to "deprefer" one provider inbound by doing AS_PATH prepending.

Although, I'm still not clear on what's causing the core issue. Why is some traffic not coming back to me regardless of which provider is preferred by the remote network?

To answer your question above - the network I mentioned is the remote network I'm trying to reach. I have a 12.x.y.0/24 that I've obtained from AT&T and they allow me to advertise this net via Sprint.

[tdrais]

Your are sorta stuck. You can balance the outbound traffic but inbound is going to be preferred on just one path and you can to a point influence that.

You can reduce the problem by taking full BGP feeds. You can hope that the closest path outbound is the closest path inbound. Unfortantly for BGP closest does not always mean fastest so you end up manually tuning this anyway.

You should not have traffic dropped because of this. It is a much bigger issue when nat is involved since that pretty much doesn't work when you have the outbound traffic different than the inbound.

I would suspect the firewall having a issue with HSRP. It is unlikely but the firewall may have something configured that is restricting the MAC address. If the firewall cares that it sent the traffic to one mac and received it back on another it may drop it.

You best bet is to turn on logging in the firewall or put a sniffer between the firewall and the outside routers. This will tell you if the traffic is even getting back to the location. I really don't think you are losing it in the internet since I see async paths all the time.

Although not very nice you could use policy routing with tracking to route the traffic that comes in from the internet on the standby router to the hot router. You would in effect be sending the traffic in the same interface the hot router was send it back out of. You may have to disable ICMP redirect since the hot router will attempt to correct this situation.

[sstone]

So, let me back up a step, do I need to specifically deprefer an ISP via AS_PATH prepending when advertising my LAN block?

I know I will need to resolve my asymmetric problem first, but once that's fixed - I'm wondering why I can't just originate my route on both ISPs and let BGP take care of itself. I realize it may not be very balanced at all, but I would hope at least both circuits would be utilized for inbound traffic without any problem.

BTW, thanks for your help.

[tdrais]

In general you are better off doing exactly that and letting BGP do its magic and see how well its working. You can always prepend later if there are issues. I guess you just get used to the Async routing strangeness, it really doesn't seem to cause much issue other than make the monitoring look funny.

The main reason I have done a prepend before was when I had a E1 as a backup to a E3.

[sstone]

Okay, again thanks for your comments.

I think I isolated the intermittient web site access problem to CBAC inspection on my MLPPP interface. It appears that it wasn't necessarily anything to do with asymmetric routing. When I removed my "ip inspect" statement, everything seemed to work fine. This is what I had configured for CBAC

ip inspect name MultiT1_Out icmp router-traffic

ip inspect name MultiT1_Out tcp router-traffic

ip inspect name MultiT1_Out udp router-traffic

When this is applied to my outbound interface, certain sites are not accessible. I'm not exactly certain what's happening. The systems that are accessing the web have full "permit ip any host x.x.x.x" on my inbound ACL (so I assumed any traffic - established or otherwise would be passed on to my internal system w/o conflicting with CBAC).

[Gajendra R']

hi ,

traffic which is orginating from your router that shuld get response back from reqested direction only. in your case outbond traffic might be going via one isp and inbound traffic reaching  from other isp.

by configuring CBAC we makeing router to do statefull inspection

Thanks

Gajendra