load balance Internet connection w/ fault tolerance?

[DANIEL WANG]

My company is researching on redundant Internet connections. We want to load balance our outgoing traffic over a cable ISP and a DSL ISP. Our DNS entries are hosted by Verisign and we hope by using IPs assigned by our ISPs we could also achieve rough load balancing for incoming traffic (which is light) through DNS Round Robin and a short TTL.

Our goal is to achieve rough load balancing with a certain degree of fault tolerance at minimal cost.

For load balancing, we found the cheapest solution is to use equal-cost default routes on the router that connects both ISPs through the ethernet ports (e0 and e1, for example).

However, using traceroute in our test lab, I found that if the link on e0 goes down, the Internet router will never switch traffic to e1. If e1 goes down, howver, router shoves data out through e0. It seems that router will always check the interface w/ lower number even when it is supposed to rotate to the other, and if the lower-numbered interface goes down, it won't bother to try the good one.

BTW, I turned off "route cache" on the interfaces. Otherwise load-balancing wouldn't even work in the first place.

How can I make the router detect the down link and switch all traffic to the other interface without using a routing protocol between us and the ISPs? We are still trying to get these ISPs to agree on exchanging routes with us. One thing I know for sure is that they don't want to do BGP for our own Class C address space.

Another scenario we are going to try is to use a route for each ISP connection and run HSRP between the two routers. Each router will be the primary of of one HSRP group and secondary in the other. Then on a 3rd router which is on the inside we put two default routes pointing to the two virtual IP addresses. By still, if a Internet link goes down while that router is alive, we will have the same aforementioned packet loss problem.

I use Cisco 4000 and 2500 in the test lab. Sorry for the windy question. Appreciate it if someone can shed some light on this problem.

daniel

[ruwhite]

For your first question, on inbound load sharing: I assume that you are running NAT on both ISP connections, so it doesn't matter which ISP a given user comes in, or that you are not running NAT at all. If you are running NAT, then you should only run it on one device, or you should run it on a pair of devices which coordinate their NAT pools.

The problem is that if a user connects inbound, and the connection is handled through one ISP, and the device answers outbound through another ISP, the IP addresses won't match because the connections are NAT'd on different devices, and the connectionwill fail.

So, with that issue aside, the idea of inbound DNS load sharing should work fine, as long as you are either providing your own DNS services, and can use some DNS implementation that supports this, or as long as Verisign provides this sort of service.

On the outbound side--how are the static routes configured? The only way the static will be removed will be if the next hop is removed from the table. If you are pointing the static default to a nexthop which is reachable through an ethernet interface (it sounds like you are?), then the nexthop won't be removed unless the interface goes down. Is the interface going down when a provider goes down, or (?).

Could you post a show ip route 0.0.0.0 0.0.0.0 when both providers are up, and when each provider is down seperately? That would help us figure out what's going on.

Finally, on load sharing.... I wouldn't turn off ip route cache on my interfaces. You definitely don't want to be per packet load sharing between two Internet Service Providers. That's just asking for trouble--probably a very low number of your sessions will actually work that way. Instead, use either per flow load sharing under CEF, which is the default, if the actual routers in the network are able to support CEF, or leave the route cache on.

In order to see the route caching working with load sharing, you're going to have to ping multiple destination addresses. Fast switching only load shares on a per destination basis--in other words, one path for each destination. If you send all of your pings to the same destination address, they are always going to travel the same path. This is by design, since packets travelling different paths in the same session will almost always arrive out of order, which will end up destroying your performance, and in many cases, will prevent the session from working properly at all.

Russ.W

[DANIEL WANG]

Russ,

Thanks so much for the great CCIE answers! I did a little study and realized I was doing it wrong in many places, exactly as you pointed out.

1) first of all, I turned NAT on both outgoing ethernet ports and configured a NAT pool and access-list for each. As a result, outgoing packets always get NATed through the pool associated w/ the lower-numbered access list. So the corresponding incoming traffic always come in through that interface. Hence the connection problem if that interface/link goes down. Here is the related IOS config on that router:

interface Ethernet0

description Cable

ip address 2.2.2.2 255.255.255.0

ip access-group Cable-ACL in

ip nat outside

media-type 10BaseT

!

interface Ethernet1

description DSL

ip address 2.1.1.2 255.255.255.0

ip nat outside

media-type 10BaseT

!

interface Ethernet2

ip address 60.0.0.1 255.255.255.0

ip nat inside

media-type 10BaseT

!

interface Ethernet3

no ip address

shutdown

media-type 10BaseT

!

ip nat pool Inside-DSL 2.1.1.100 2.1.1.200 prefix-length 24

ip nat pool Inside-Cable 2.2.2.100 2.2.2.200 prefix-length 24

ip nat inside source list 10 pool Inside-Cable

ip nat inside source list 11 pool Inside-DSL

ip nat inside source static 60.0.0.10 2.2.2.10

ip classless

ip route 0.0.0.0 0.0.0.0 Ethernet0 2.2.2.1

ip route 0.0.0.0 0.0.0.0 Ethernet1 2.1.1.1

no ip http server

!

!

ip access-list extended Cable-ACL

permit ip any host 2.2.2.10 log

deny tcp any any log

deny udp any any log

remark this is a named extended ip ACL

permit icmp any any log

access-list 10 permit 60.0.0.32 0.0.0.31

access-list 11 permit 60.0.0.32 0.0.0.31

access-list 100 permit ip any host 2.2.2.10 log

access-list 100 deny ip any any

So if I really want to load balance outgoing tranffice thru the two ISP connections, I have to NAT both connections and have to use two devices, right? Why do I need to make the two device co-ordinate their pools. How?

2)>The problem is that if a user connects inbound, and the connection is handled through one ISP,

>and the device answers outbound through another ISP, the IP addresses won't match because the

>connections are NAT'd on different devices, and the connectionwill fail.

How do we fix this? If I have two devices connecting to the two ISPs, then I need a 3rd piece on the inside to load balance traffic to the two NAT devices. How would the inside device know to route the response back to the same NAT device through which the original incoming packets travel?

3) All my load balance problems with the static default routes are found to be related to (static)routing problem in the test lab. I read up on IP switch and, you are right, I should turn on CEF per flow.

4) All the above came from my first test scenario -- using one internet router connecting to two ISPs with two static default route for load balancing. The inside interface then connects to our two redundant PIX 515e.

5) If Setup #1 won't work, I am thinking about #2 -- using 3 routers. One for each of the ISP connection and NAT, the 3rd one on the inside (actually, before the PIX) for load balancing. And instead of using HSRP and static default routes, I might just need to configure a static default route on each NAT router, turn on EIGRP or OSPF on all three, and the 3rd router should be able to load balance using the equal-cost default routes redistributed through the routing protocol. If one NAT going down, the corresponding default route should be gone on the inside route, hence the fault-tolerance.

Lastly, in either setup, the NAT router does not connect directly to the ISP routers but thru their modems. So unless I can persuade the IPSes to run a routing protocol with us, there is no way my routers will find out when our uplinks go down to adjust routing accordingly.

The only way I can think of right now is to write a ping script and install a network monitoring problem that constantly pings our near end and far end routers of our ISP (and I need to configure static routes to these routers to avoid load balance). If we receive an alert from the monitor, we then would have to manually adjust routing on our router.

Many thanks for bearing with me for such a long one :)

daniel

[ruwhite]

> 1) first of all, I turned NAT on both outgoing

> ethernet ports and configured a NAT pool and

> access-list for each. As a result, outgoing packets

> always get NATed through the pool associated w/ the

> lower-numbered access list. So the corresponding

> incoming traffic always come in through that

> interface. Hence the connection problem if that

> interface/link goes down. Here is the related IOS

> config on that router:

I actually thought you were running NAT on two devices, not suggesting that you do that. I assumed you were running NAT on the two PIX's, which would be much harder than NAT'ing before you get to the PIX. :-)

It sounds like your second provider isn't routing traffic through to you if the first provider link fails--so what you'll need to do is to get each provider to advertise your address space from the other provider. For instance, if you received 10.1.1.0/24 from provider A, you need to get provider B to advertise that space as well. And if you got 10.1.2.0/24 from provider B, you need to get provider A to advertise that space as well. That way, when the link to either one of them fails, the other one will be able to route traffic back into you, and get it back to the NAT router.

I'm not certain how to handle the problem with NAT on the inside router always taking the lower numbered pool--I'll have to ask someone else about how to handle that, or if it can be solved. I'll get back to you on it when I hear from him, or let him answer here directly. Your best solution is bound to be the first one, rather than running NAT in two places, if we can make that work.

Russ.W

[ruwhite]

I asked one of the NAT experts (Mike Sullenberger) around here for help, and I'll just paste the reply in below....

The above configuration will not do waht you want. When NAT needs to create a

new translation it will always search through the 'ip nat inside source list

...' commands in the order they are configured. Therefore you will always match

on list 10 and use pool Inside-Cable. If you want to select the NAT pool by

outbound interface you need to use the following configuration.

ip nat inside source route-map Inside-Cable pool Inside-Cable

ip nat inside source route-map Inside-DSL pool Inside-DSL

route-map Inside-Cable permit 10

match ip address 10

match interface Ethernet0

route-map Inside-DSL permit 10

match ip address 10

match interface Ethernet1

This will load balance the _creation_ of the NAT translation entries per

flow across the two pools. This will not guarantee that the NATted

packets will actually be forwarded out the interface that a particular

pool is associated (in the route-map). After the initial NAT translation

entry is created there is NO communication between NAT and

routing/forwarding, the two operations are independent of each other.

If you are process-switching then packets NATted with either pool will

go out either interface on a packet by packet basis.

If you are fast-switching then packets within a flow that are NATted

with a pool will continue to use the same outbound interface until the

Fast-switch cache entry is refreshed. At that point the flow could stay

with the same interface or it could switch to the other interface.

If you are CEF-switching then packets with the same source/destination

address should be NATted with the same pool and will continue to use the

same outbound interface.

In all cases if you lose one of the outbound interfaces then all of the

packets will be sent through the other interface, but NAT will still use

the same translation entries. NAT must work this way, you cannot have

NAT change the NATted source address in the middle of TCP connection.

If your ISPs will not allow packets sourced from the other ISP's address

space then there is not much that you can do. There will always be a

situation where connections will be broken because packets NATted with

one ISP's address space will go out the interface for the other ISP.

CEF-switching is the closest you can get to having the least amount of

broken TCP connections, but it won't be perfect.

:-)

Russ.W

[DANIEL WANG]

Russ,

Great explanation! I thought you were too busy to notice my last post :)) I just found your post two days ago and implemented rout map and it worked great! Many thanks to you and Mike for the wonderful NAT answer. I also found a way to policy route our public web server's (which has two NATed outside addresses) http responses back out the responding ISP links by assigning to the web server two private IPs which are statically NATed to the two outside addresses respectively.

As I am wrapping up this project (thanks to your help), I realize that I've got to find a solution or workaround to the problem in your last paragraph --

if my ISPs "will not allow packets sourced from the other ISP's address space" and "there will always be a situation where connections will be broken because packets NATted with one ISP's address space will go out the interface for the other ISP."

So if one ISP link goes down, all our sessions thru that link will break because they will have to go out the other ISP while carrying a NATted source ip for the broken ISP. Users of the broken sessions will continue to experience new connection problem because packets from the new connection will still be NATed for the broken ISP side before their old dynamic NAT xlate entries time out.

Is there a way for the route to clear these old NAT translations? If not, is there a scripting way that works with IOS to allow me clear those old NAT translations automatically upon learning about the broken ISP link? I heard the new IOS version will have the scripting capability like Unix. That's great! But what about now?

[ruwhite]

"So if one ISP link goes down, all our sessions thru that link will break because they will have to go out the other ISP while carrying a NATted source ip for the broken ISP. Users of the broken sessions will continue to experience new connection problem because packets from the new connection will still be NATed for the broken ISP side before their old dynamic NAT xlate entries time out."

Kindof.... You shouldn't lose all your connections until the old NAT translations time out, you should just lose your connection, and it will work the next time you try. In this configuration, you're also tying the translation table used to the outbound interface, correct? So, if a packet arrives and is routed out interface A towards ISP A, it will be assigned a source address from pool A. Likewise for interface B, ISP B, and pool B. So, suppose the link to ISP A fails. The next packet that comes in will be routed out interface B, to ISP B. This means that it will be assigned a source address from pool B.

Now, when the host on the other end of the network gets this packet, it will not know what to do with it--it will look like a packet from the middle of a seesion that it doesn't have any record of at all. So, it will drop it, and not ack it. The originating host will retransmit a couple of times, and then break the session. Once this happens, if the user attempts the connection again, the session will start with the address from pool B, and everything will work.

It doesn't matter that there is an old translation in the table, since the traffic isn't being routed through interface A, and that pool will only be used for traffic that's routed through interface A. Now, as to how to solve even this disconnect, there's really only one way that I know of, and that's to advertise your address space to both ISPs, etc.

:-)

Russ.W

[DANIEL WANG]

Thanks for the quick reply! But I disagree with you on how NAT works for new connections after ISP A goes down. From my reading of Mike's reply as well as CCO site, I think as long as there is an existing NAT translation for an inside local address, the router will continue to use it for any packets (incld. new connections) from that inside host until the NAT translation times out. In this case the default timeout is 24 hours for dynamic NAT of TCP sessions. So I have to "ip nat translation tcp-timeout N", hoping N seconds will be short enough to clear that old translation.

So if an inside client's connection to yahoo.com through ISP A is broken and is now redirected through ISP B, the NATted source ip will still be the same, causing the connection to break. Now as long as the client doesn't keep trying within N seconds, the old NAT xslate will timeout. So the new connection from the same client to Yahoo will get a new NAT according the route-map tying NAT pool of ISP B to the interface. Fortunately, most of our traffic going out is SMTP and HTTP, so a N = 30 seconds timeout may be good enough to clear the old translation when ISP A breaks. Here is a quote from Mike:

"This will load balance the _creation_ of the NAT translation entries per

flow across the two pools. ****** This will not guarantee that the NATted

packets will actually be forwarded out the interface that a particular

pool is associated (in the route-map). After the initial NAT translation

entry is created there is NO communication between NAT and

routing/forwarding, the two operations are independent of each other. ******

....

If you are CEF-switching then packets with the same source/destination

address should be NATted with the same pool and will continue to use the

same outbound interface.

In all cases if you lose one of the outbound interfaces then ******* all of the

packets will be sent through the other interface, but NAT will still use

the same translation entries. NAT must work this way, you cannot have

NAT change the NATted source address in the middle of TCP connection.**********

Well. I got a more serious problem, here. For whatever reason, using route-map to tie an interface to a NAT pool didn't work for me. I followed Mike's example to the letter and checked CCO documentations. I even rename the route-map name differently from the pool name. Just didn't work -- my inside outgoing traffic is NOT NATed at all!

Interface Ethernet 0

Description Link-to-Cable ISP

IP Address ....

IP NAT outside

Interface Ethernet 1

Description Link-to-DSL ISP

ip address ....

ip nat outside

Interface Ethernet 2

Description Corporate

...

ip nat inside

....

ip nat pool Inside-Cable ...

ip nat pool Inside-DSL ...

ip nat inside source route-map Cable pool Inside-Cable

ip nat inside source route-map DSL pool Inside-DSL

route-map Cable permit 10

match ip address 10

match interface Ethernet0

route-map DSL permit 10

match ip address 10

match interface Ethernet1

access-list 10 permit ...

However, if I remove "match interface ethernet ..." line from both route-maps, the traffic will NAT. Of course the NATted addresses are not tied to a specific interface/ISP route this way.

Once I put the "match interface" statement back, route won't NAT the traffic. But according to IOS mannual and CCO examples, this command should tie new NAT translations to interface to the next hop in the route.

Am I missing something here? Thanks for your patience.

daniel

[DANIEL WANG]

Upon further testing. It turned out ICMP packets are going straight out w/o NAT (I have not put a ACL on the outside interfaces yet). I was testing using Ping and Traceroute. When I switched to telnet, NAT and route-map worked, even better than I had expected -- it created a new translation for a new connection out ISP B when ISP A goes down.

The reason? Because I am using route-map for NAT, now the NAT table becomes an extended table vs. one created by Access List.The extended trans. table includes port numbers for the address pairs. So every new connection is a new translation! I don't have to worry about the timeout now!

Now my new problem is, how to make ICMP packets go NATted, which should not happen according to Cisco Docs and makes no sense any way.

Isn't it funny every time I think I've got it solved there is always another popping up?

I hope I haven't bug you too much. I assure learned a lot from you.

:))

[ruwhite]

I would think that ICMP packets would be NAT'd--it seems to say so here:

http://www.cisco.com/en/US/tech/tk648/tk361/technologies_white_paper09186a0080091cb9.shtml

What do you see that's convincing you it's not covered?

:-)

Russ.W

[DANIEL WANG]

Until Friday, I was basically testing w/ ICMP and protocol analyzers. Ethereal traces on the dest. host showed that the icmp pings were sourced from the internal IP, not the NATted one. That's why I though the route-map didn't work. Then I changed to Telnet, the packet capture showed that they were NATted correctly. But the ICMP packets still weren't. This doesn't make sense. I will further test it today.

One good thing that came out of this, though, is we don't have to worry about the persistent old NAT translation any more. NAT based on route-map creates an extended transaltion table which pratically guarantees a new translation for each new session!

Will read up on the white paper and keep you posted.

:))

daniel

[shechter]

I use Linkprof from Radware.

It is simple, and it is working.

You can LB your outgoing traffic and you incomming traffic, even if you have several different external addresses from several isp's.

Dan

[DANIEL WANG]

I looked at the web site. I wander what the pricing is. We didn't look at products like this because we always invested in Gigabit switchs, PIXs, VPN, IDS, and anti-virus management. We are just trying to be cheap after spending half a million dollar upgrading our network :))

And I believe our current Cisco routers can do it in IOS. Just have to work the kinks out.

Thanks,

daniel

[ryang]

I look at a product like radware or the el-cheapo nexlan routers that can basically do automated ping tests to determine whether an upstream provider is down, since Cable and DSL providers are extremely unlikely to feed you a routing protocol. I wonder if cisco will ever support something like this?

I've implemented per session load balancing with NAT and two providers much like discussed in this thread, but if one provider goes down, the ethernet link is still 'up' as far as the cisco is concerned, but it will still blindly send packets out that interface....

[ruwhite]

There is a product coming out in the future that will be able to monitor the link, and switch to another path if one fails, but it's not based on ping tests. It's called Optimized Edge Routing. I think we also considered a static which is verified with a periodic ping, but I'm not certain if this was ever implemented, or even how (really) useful it would be.

:-)

Russ.W