Ability to accept traffic from two ISPs to same host

t.khan · ‎06-17-2010

Hello,

I am stumped on how to configure a network to be able to accept traffic from two ISP's to one host.

We have two ISPs, connected to two firewalls, each firewall has a static NAT mapping for the external IP to an internal IP

ISP1 --> FW1 NAT externalip1 ->192.168.100.10

ISP2 --> FW2 NAT externalip2 -> 192.168.3.10 --> rtr2 NAT --> dest ip 192.168.100.10

Then we have a router behind each firewall, network2 the router changes the destination address from 192.168.3.10 to 192.168.100.10. The packet then gets routed over the internal network to host 192.168.100.10. Two problems occur:

1) The host 192.168.100.10 sends the return traffic out its default gw, which sends to the fw, which attempts to send out gw of isp1 (even though traffic originated through isp2. Is there a way to create a route-map that sends the traffic back out to fw2/rtr2?

1a) Could we NAT the external IP from isp2 to an inside address and use the routing table to send the traffic back to rtr2?

2) The traffic (since it is external) gets dropped by the firewall because of spoofing. This occurs because the firewall is not expecting an outside (internet) address to come in on that interface. we have relaxed the anti-spoofing rules to circumvent, but the routing issue still occurs.

Currently, we have this setup as a failover solution. If ISP1 goes down, we just route all traffic through the network out ISP2. We would like to load balance and use both ISP connections.

We also have p

Any ideas?

bshellrude · ‎06-17-2010

Hey T.khan

Well, the way I see it you have a a couple of options, some or none of which may be viable depending on your setup/capabilites of your network...

The problem you're encountering is that your network has no way to differentiate between the two ISP's in terms of what networks are closer/shorter-path through which ISP. In other words, if I'm on ISPB, with IP 10.10.10.1, because your external IP 2.3.4.5 is owned/routed-by ISPB, my path to you will be via that connection. However, on the return path, your network has no way of knowing that 10.10.10.1 is closer through ISPB than it is through ISPA...

Easiest way to have the return traffic take the same path as it did inbound, would be to see if your providers would be willing to peer with you (BGP) and send you full internet routes. Keep in mind you'll likely need a router with at least 512Mb memory to store the 150k+ routes that exist on the internet today. Though you "could" use linux as a router which may be a cheap alternative to say a 7604...

Second option might be, that while it won't solve the problem of keep symmetrical paths between IP Pairs, you could still load balance traffic. You could set two equal cost 0.0.0.0/0 routes on the network. Though this may cause problems depending on the hardware you're using. Typically with CISCO CEF, if you set a per-destination load blancing, for equal cost routes, each "IP Pair" will load balance. However, you may end up in a situation (depending on how your firewalls do it), where you get a "per-packet" load balancing... which might make things worse than better... i.e. packets are received out of order/sequence by either end, incurring tcp retransmission, essentially slowing the connection...

Third option might be to see if (and there's a very slim chance of this being a possibility), but if you could find some portable address space, or if one of your ISP's could provide you with say a /27 or something of portable address space and you could get the other ISP to agree to accept an advertisement for that block, then you could use a combination of the first option and this, and achieve a true load balancing based on BGP best path selection, as now users around the world would take the "best path" to get to you, and vica versa...

On the other hand... another option, which may also work, would be to dual home the box. One to each firewall. (actually this may be the simplest) In theory, traffic received on a given IP, should leave via that same ip, thus taking whatever default gateway the next hop (firewalls) are using... Come to think of it, this may be the best/easiest/least complicated of them all... You'd have to set a default gateway (all 0's route) via both firewalls too (i.e. default gateway for each interface)...

Know this may not be the news you wanted to hear... but figured I'd throw my 2 c out there....

Cheers,

Ben

t.khan · ‎06-17-2010

Ben,

Thanks for the reply. Those solutions should work, but unfortunately I don't think we can implement them.

Do you know if we can NAT the external source (from internet) IP to an internal and then let the router route back to that network?

[SA-1.2.3.4]-[DA-67.12.3.3] --NAT(RTR1)--> [SA-1.2.3.4]-[DA-192.168.200.10] --NAT(RTR2)--> [SA-192.168.3.x]-[DA-192.168.200.10]

That way when 192.168.200.10 responds, he replies back to 192.168.3.x which is on the other network. Will that work?

What about using a route-map (I think) to specify if traffic from destination x came on interface y, make sure to set the next hop back to int y and not use the def gw?

Does this sound even remotely possible?

bshellrude · ‎06-17-2010

Yeah... you know actually SNATing the sources to a local IP specific to each firewall would work!! Though... if this is a web server or something similar where you want to track who's hitting the site, those stats will go out the window obviously as all you'll see is the SNAT addy's...

The route-map... slim chance it might work... you'd have to try it. You're talking about PBR (Policy based routing), which in all my experience I've used based on src/dst networks... i.e. if coming from src x and destined for network y, next hop z.... Someone else may chime in here that's done it based on interface... It would definately be worth a try if you had a maintenance window you could play around in...

Cheers

t.khan · ‎06-17-2010

Ben,

Thank you again.

Now for the real question?

How to I get he source NAT to work? I have tried it on RTR1 and RTR2, but I must be doing it wrong. I set up an ACL that basically said any traffic to 192.168.100.10 and then deny all.

The NAT was supposed to grab any grab any traffic to 100.10 and NAT the source, but it failed.

Any ideas?

bshellrude · ‎06-17-2010

What kind of firewalls?

t.khan · ‎06-17-2010

The FW's won't let me change the Source Address.

Can the a 3845 do it once it hits the inside Router?

5.6.7.8 is an externally addressable IP

1.2.3.4 is the client IP on the internet

192.168.0.10 is the NAT'd IP that the router is proxy ARPing for

192.168.100.10 is the destination IP on network 2

The External FW NATs like this: [SA-1.2.3.4]-[DA5.6.7.8] --NAT--> [SA-1.2.3.4]-[DA-192.168.0.10]

The Internal Cisco Router NAT's: [SA-1.2.3.4]-[192.168.0.10] --NAT--> [SA-1.2.3.4]-[DA-192.168.100.10]

The packet is then routed to RTR 2, who routers it to FW2.

Can I on RTR 1 NAT the SA as well? So it does this:

The Internal Cisco Router NAT's: [SA-1.2.3.4]-[192.168.0.10] --NAT--> [SA-x.y.z.z]-[DA-192.168.100.10] where x.y.z.z is an address that is routable back to RTR1?

bshellrude · ‎06-17-2010

edited

Message was edited by: bshellrude

bshellrude · ‎06-17-2010

edited

Message was edited by: bshellrude

bshellrude · ‎06-17-2010

Heh... even that link won't work.

So lets do this another way.

If you consider your inside interface (where you want your SNAT to be located) as your "outside interface"

And your outside provider-facing interface as your "inside" interface

ip nat outside source static x.x.x.x 192.168.100.10 (where x.x.x.x is the Ip you want to have traffic nat'd to)

Think that should work...

or never mind... I don't think it will. The tricky part is we don't want everything to be nat'd otherwise we'd just do a hide nat...

What you can probably do instead is:

ip nat pool SNAT x.x.x.x x.x.x.y netmask 255.255.255.0 (where x.x.x.x-x.x.x.y are the addresses you want to have source traffic nat'd to)

ip nat inside source list 10 pool SNAT

access-list 10 deny i.i.i.i n.n.n.n (where i.i.i.i and n.n.n.n signify your internal networks/masks you DON'T wan't nat'd)

access-list 10 permit any

There... THAT should work

Message was edited by: bshellrude

t.khan · ‎06-18-2010

Ben,

I actually had tried what you suggested earlier. As crazy as this sounds, it starts to work, but then the host 192.168.100.10 actually tries to return the SYN-ACK through the original route. I am beyond stumped. What I am doing below is just a POC, I am only testing from one server in the x.y.z.a subnet.

The NAT Rule is:

ip nat inside source list ACL_INT2_ANY interface FastEthernet0/0 overload

ACL:

ip access-list extended ACL_INT2_ANY

remark #######################################################
remark Permit interesting traffic ISP2 to WebServer

remark #######################################################
remark
permit ip x.y.z.a 0.0.0.31 HOST 192.168.100.10 log
deny ip any any

As crazy as this sounds, the External Client (me for testing) from the internet connects to ISP2 External NAT addr, that gets routed over to RTR2 and hide nats behind INT FE0/0, the webserver sees traffic originating from FE0/0 to 192.168.100.10, then, it tries to reply to the proper IP Addres (The Source that I am supposed to be hiding behind FE0/0.) I am tcpdumping traffic off of the fw, and ip nat debugging the rtr to find out how in the hell that is happening. I am stumped as to where the 192.168.100.10 server even knows about the external IP that is supposed to be hidden behind FE0/0.

Basically I see a SYN packet from FE0/0 to 192.168.100.10, then 192.168.100.10 sends a SYN-ACK to x.y.z.a. Where does it get that IP?!?!? ( have verified it is the same handshake based on the source port being the same each time)

The only think I can think of is that RTR2 is trying to send out traffic his def gateway and not the return path.

I will debug further and see what I can find out.

Thanks for all your help.

t.khan · ‎06-24-2010

Finished....

The NAT rule that modified packet x.z.y.a --> 192.168.100.10 to FA0/0-overload --> 192.168.100.10 worked fine, but when the router took the reply packet and replaced the NAT address, the destination address was x.y.z.a which is external, so the router sent it out the default route. I got a wonderful assymetric routing issue. I finally fixed it, I created a route map that when a packet with source 192.168.100.10 --> x.z.y.z was seen, sext the ipnext hop to the other router RTR1.

Rules for other ppl:

NAT RULE:

ip nat inside source list ACL_GCIN_ANY interface FastEthernet0/0 overload

ACCESS LISTS:

ip access-list extended ACL_GCOUT_ANY
remark #######################################################
remark Permit interesting traffic from GC out Sterling
remark #######################################################
remark
permit ip host 192.168.100.10 x.y.z.a 0.0.0.31 log
deny ip any any
ip access-list extended ACL_GCIN_ANY
remark #######################################################
remark Permit interesting traffic from sterling to GC
remark #######################################################
remark
permit ip x.y.z.a 0.0.0.31 host 192.168.100.10 log
deny ip any any

Route MAP

route-map RMAP_x.y.z.a permit 10
match ip address ACL_GCOUT_ANY
set ip next-hop 172.16.2.2

INTERFACE:

ip policy route-map RMAP_x.y.z.a

Glenn R · ‎09-07-2010

Perhaps you can help me I have a similar issue to you and don't seem to understand how you solved this.

Routing to default gateway when the traffic cam from a different firewall

My configuration is

ISP1 -- --> FW1 -- --> RTR1 -- --> L3 SWITCH <-- -- -- -- -- -- --> SERVER1

|

ISP2 -- --> FW2 -- -- -- -- -- -- -- -- -- -- -->

IP Addresses ( examples )

ISP1 - 212.10.10.0/27

ISP2 - 208.10.10.0/27

FW1 out - 212.10.10.1/24

FW2 out - 208.10.10.10/24

FW1 in - 192.168.0.10/24

FW2 in - 192.168.1.10/24

RTR1 inside - 192.168.1.11/24

RTR1 outside - 192.168.0.11/24

SERVER 192.168.1.220/24

SWITCH is a layer 3 switch with multiple VLANS and route-map policies to route between VLANS and choose gateway based on VLAN.

The router has NAT enable and a translation for 192.168.0.220 --- static map to --> 192.168.1.220 with a machine in the 192.168.0.x netork I can view a test web server on 192.168.1.220. Anything outside FW1 cannot get the web server even with all static and nat command working.

Please coudl someone helpme with this

G