Solved: GLBP working, now want interface redundancy?

BradenWright · ‎12-07-2011

So I have 2 routers (cisco 3640) that each go to their own ISP and then back to the same switch. I have setup ospf and glbp, and now have pretty good redundancy. If either internet connection or routers go down everything is still golden.

So I was thinking that if an interface went down then the router would not be load balanced with glbp which got me thinking whats the best way to get interface redundancy (and I was going to add a 2nd switch with the second interface)... my thoughts

1) Setup BVI on the 2 interfaces.

2) Setup a 2nd interfaces (on each router), I would have to split the subnet, for instance:

RouterA

IntFa0/0

192.168.0.1/24

IntFa0/1

192.168.1.1/24

then the machines could be on the subnet 192.168.0.0/23 and setup glbp for 1 ip across all 4 interfaces (I'm not even sure if you can do this but think it would work).

3) Is there a way to utilize etherchannel or anything like this

A negative to option 2 would be that if 1 of the interfaces went down, all the sudden 2/3 (or so) of your traffic would be going through 1 router

Just looking for the best way... I may not have even touched on the right solution... I'm open to any solution... was leaning towards trying BVI but saw a post where someone said its messy.

Jon Marshall · ‎12-08-2011

Okay a few points to consider -

1) if you have a pair of firewalls and run them in active/standby and insert them between the routers and the internal switches then GLBP gives you nothing. This is because the active firewall will be the only mac-address seen by the routes and so will be assigned to the same router every time. So you find that only one ISP would be used unless the router failed and then it would be the other ISP.

2) If you do insert the firewalls then you would use a switch between the routers and the firewalls. This means that if a LAN interface on one of the routers fails then the firewalls won't know about it because their connection to the switch is still up. The ways round this are -

1) use IP SLA on the firewalls if they are ASA

2) use a dynamic routing protocol between the routers and the firewalls ie. you could run OSPF between your routers and firewalls.

3) Not clear on whether you want to use both ISPs simultaneoulsy or one and then the other if the first fails. If you want to use one and only the other as backup then you need to influence the static route so one is favouted over the other.

4) Not many people worry about interface failover in the way you are. If you have 2 of everything including internet connections then you have the redundancy. The whole idea of this design is that you have removed any single point of failure.

If you do want it there are a couple of options -

i) BVI as mentioned

ii) use a L3 switch between the firewalls and the routers. Use 2 LAN interfaces on each router and connect back to the switch. These would be on different subnets. Then the firewall would also connect into this switch on a different subnet again.

You then run a dynamic routing protocol between the firewall(s) and the routers.

So each router will advertise it's route(s) via both LAN interfaces. The firewall then sees 2 equal cost paths to each router so in effect it gets 4 routes (2 from each router).

Drawback to this is that using a L3 switch on the outside of your firewalls is often frowned upon.

iii) You could directly the routers and then if the LAN interface of one router fails there is still a path to that router. But the problem here is that for outbound traffic it would never go across that path because the router with the active LAN interface has a direct route to the internet.

However where it is useful is if you are presenting servers to the internet and you do not have provider independant addressing ie. you use one of the ISPs blocks for servers. If the router that is connected to this ISP (R1) looses it LAN interface the block could still be advertised and traffic could come to R1 and then be routed across the interconnect to the other router and down to the firewall(s).

There is so much that could be covered here but it is going to confuse things even more. As you say you still have a lot of things to tackle and i would recommend concentrating on designing/configuring this before you worry too much about interface redundancy. It's not that important compared to all the other things you need to get working.

Things to be aware of -

1) ISP addressing. As you can see from the above the addressing can have a large effect on how the design works. If you have 2 blocks, one from each ISP this can affect how you advertise out servers within your network

2) firewalls - if you are looking to use firewalls then you need to decide which ones, are they going to be running in active/standby, active/active etc. Again how you run them has a big effect on the design. Bear in mind with firewalls between your internal switches and routers you probably won't be running GLBP on your routers.

You also need to make sure any firewall you purchase has the functionality you need.

3) ISP - one or 2. If you use one ISP you could have multiple circuits off them. Certainly in the UK we could order 2 circuits from the same provider to the same site but with completely different routing of the circuits. It may make things easier to begin with.

It sounds like you are going from a no redundancy situation so it can be done in steps rather than try to do the whole thing at once. Certainly the design should be flexible enough to add more redundancy but it might not be advisable to do the whole thing at once in terms of implementation. I am not trying to put you off at all but there is a lot to think about in these setups.

Please feel free to come back with further questions as you may well have more.

Jon

View solution in original post

Jon Marshall · ‎12-07-2011

Braden

A few clariifications would help.

1) you say you have OSPF and GLBP so where is OSPF running between - is it between your routers and the ISP routers as this is slightly unusual.

If not where is OSPF running ?

Do you run any dynamic routing protocol between your routers and the ISPs ?

2) the 3640 routers. Each has a connection to it's relevant ISP and a connection back to a common switch ? Is that correct. ?

If so, is there also a direct connection between the 2 routers ?

3) the switch - is it just a layer 2 switch with end clients connected into it ? If not what is the setup behind the 3640s ?

4) Is there a firewall(s) anywhere in this topology ?

5) If you have 2 ISPs then do you have your own public IP block ie. provider independant or do you have 2 blocks, one from each ISP ?

If you do have 2 blocks outbound internet access is not a problem but do you host any servers that are accessible from the internet ?

6) where is the NAT taking place, assuming you are using private internal addressing.

Apologies for all the questions but to suggest a solution we really need to understand the topology and routing of what you currently have.

Jon

BradenWright · ‎12-08-2011

AHHHH... I'm just in lab/testing phase right now....

So technically there is no ISP I am just simulating them, so some of these questions don't really apply right now. I was out of town and hoping to do some research on the best way/technology to solve the issue. So I could start playing next week.

I can post a little pic and my 2 router configs (next week when I'm back in town) if you want, but....

1) I am running OSPF b/t the 2 routers. Each has a static default route (which is going to my fake ISP) and I am redistributing the static route so if 1 of the internet connections goes down, everything starts going through the other routers Internet connection.

2) You are right, right now no direct connection b/t the 2 routers but I was leaning towards adding one (most setups I have seen have one).

3) I have 2 switches which I can play with 1 is Catalyst 2950 (L2) one is a Cisco SG300 (which is L3), although right now I am using the Layer 2 switch

4) No firewall right now (I will be adding one or 2) but haven't got there yet. (If you have a fairly inexpensive reccomendation I would be open. Currently we have a checkpoint linux box, and I'm not very impressed)

5) Not applicable yet... but I was probably leaning towards 2 seperate ISPs and checking out BGP (but outside the scope of what I'm looking at now)

6) Right now PAT is being used on the routers (but again not sure it will stay there, just how I have it now for testing).

Basically right now we have a single point of failure for just about everything. We have 1x 3640 routers which goes to our checkpoint firewall (which does NAT/PAT) and that goes to our primary switch. If we have an Internet failure... someone manually changes the cables to move use over to our other line. Since I've come on board, I was like are you joking?!?!?

I am trying to redesign the network to have redundancy, so the next step to me, is interface redundancy, so try to figure out the best technology to use (or options).

After that as you have mentioned I still have a lot of things to tackle like firewalls, switches, incoming redundancy via bgp or using a load balancing appliance, etc.

Jon Marshall · ‎12-08-2011

Okay a few points to consider -

1) if you have a pair of firewalls and run them in active/standby and insert them between the routers and the internal switches then GLBP gives you nothing. This is because the active firewall will be the only mac-address seen by the routes and so will be assigned to the same router every time. So you find that only one ISP would be used unless the router failed and then it would be the other ISP.

2) If you do insert the firewalls then you would use a switch between the routers and the firewalls. This means that if a LAN interface on one of the routers fails then the firewalls won't know about it because their connection to the switch is still up. The ways round this are -

1) use IP SLA on the firewalls if they are ASA

2) use a dynamic routing protocol between the routers and the firewalls ie. you could run OSPF between your routers and firewalls.

3) Not clear on whether you want to use both ISPs simultaneoulsy or one and then the other if the first fails. If you want to use one and only the other as backup then you need to influence the static route so one is favouted over the other.

4) Not many people worry about interface failover in the way you are. If you have 2 of everything including internet connections then you have the redundancy. The whole idea of this design is that you have removed any single point of failure.

If you do want it there are a couple of options -

i) BVI as mentioned

ii) use a L3 switch between the firewalls and the routers. Use 2 LAN interfaces on each router and connect back to the switch. These would be on different subnets. Then the firewall would also connect into this switch on a different subnet again.

You then run a dynamic routing protocol between the firewall(s) and the routers.

So each router will advertise it's route(s) via both LAN interfaces. The firewall then sees 2 equal cost paths to each router so in effect it gets 4 routes (2 from each router).

Drawback to this is that using a L3 switch on the outside of your firewalls is often frowned upon.

iii) You could directly the routers and then if the LAN interface of one router fails there is still a path to that router. But the problem here is that for outbound traffic it would never go across that path because the router with the active LAN interface has a direct route to the internet.

However where it is useful is if you are presenting servers to the internet and you do not have provider independant addressing ie. you use one of the ISPs blocks for servers. If the router that is connected to this ISP (R1) looses it LAN interface the block could still be advertised and traffic could come to R1 and then be routed across the interconnect to the other router and down to the firewall(s).

There is so much that could be covered here but it is going to confuse things even more. As you say you still have a lot of things to tackle and i would recommend concentrating on designing/configuring this before you worry too much about interface redundancy. It's not that important compared to all the other things you need to get working.

Things to be aware of -

1) ISP addressing. As you can see from the above the addressing can have a large effect on how the design works. If you have 2 blocks, one from each ISP this can affect how you advertise out servers within your network

2) firewalls - if you are looking to use firewalls then you need to decide which ones, are they going to be running in active/standby, active/active etc. Again how you run them has a big effect on the design. Bear in mind with firewalls between your internal switches and routers you probably won't be running GLBP on your routers.

You also need to make sure any firewall you purchase has the functionality you need.

3) ISP - one or 2. If you use one ISP you could have multiple circuits off them. Certainly in the UK we could order 2 circuits from the same provider to the same site but with completely different routing of the circuits. It may make things easier to begin with.

It sounds like you are going from a no redundancy situation so it can be done in steps rather than try to do the whole thing at once. Certainly the design should be flexible enough to add more redundancy but it might not be advisable to do the whole thing at once in terms of implementation. I am not trying to put you off at all but there is a lot to think about in these setups.

Please feel free to come back with further questions as you may well have more.

Jon

BradenWright · ‎12-08-2011

Thanks for the info, it really helps.... you are exactly correct we have no redundancy right now I am trying to build it up.

1) I get what you are saying about the firewall. I felt I didn't have enough info to make the decision (right now to play with I have some linux boxes I can setup or we have a couple old pix fws, I have not messed with pix at all). I do have some room to make decisions, I was leaning towards active-active firewalls (mainly b/c I normally feel if you have the hardware might as well use it, plus with monitoring you can basically guarantee your failover will work). But curious opinions (lower cost would be preferrable)

2) So I was going to start by no having a switch b/t the routers and firewalls and then would move on to trying to add one (I guess how I do that would depend on the firewall, so thanks for your info, this is the reason I decided it was better to go with ospf over eigrp).

I take it adding a switch inbetween and using routing protocols would allow for further control and expandability to scale out the network horizontally (fw's and routers)???

3) I would like to use both ISP's similatenously b/c if I can then it's easier to justify a bigger/equal second line. (Same with incoming traffic but thats a whole another beast beyond this scope). Right now we have a crappy back up line, and I have basically setup a second 3640 router which goes to it (and with static routes and ospf as you have mentioned I have failover going, so that was at least a step forward now looking to beef that up to an active-active b/c the backup line isn't enough to run the whole office).

4) Thanks I will forget about interface failover on the router I know they are normally pretty rock solid. And get what you're saying

Some of the other things you mentioned at the end. You are definitely right on, I think my next step is to figure out what firewall I am going to get. Then after getting the firewalls up and doing some testing I can figure out the ISP? With that I was leaning towards 2 different providers (but the only reason was so I could try to make sure the routes where as seperate as possible, which I'm not even sure if getting separate providers helps this). I really I didn't have a ton of reasoning (I guess another benefit might be for further scale out might be easier if it were ever needed).

Side Question: If glbp won't work to load balance between isps... if I have something like ospf running between the firewalls and routers (since I think I can set this up in my test environment and it will probably be my next step as I start to look at some firewalls) would you just use ospf to load balance?

[Little Back Story: Basically other than our network We are in pretty good shape. We have generaters/battery backups we virtualized most our servers and our managers wanted to start talking about DR (we host our own email being most critical but rent server space for some websites). I was thinking we were putting the cart before the horse, and that we should first get redundancy in our office. So I have some experience with Cisco (probably b/t CCNA-CCNP) but its been a few years and all of it was in the lab (so not much with firewalls/live isps etc). I am hoping that if I can provide enough redudancy we can bring some of our web servers inside our network. But just trying to build up 1 step at a time (but want it to be built intelligentally so it can expand) I'm sure DR site will be next.]

THANKS SO MUCH! You brought up a few points I hadn't considered!

Jon Marshall · ‎12-08-2011

1) Only used checkpoints and ASAs. One thing to be aware of with ASAs is that active/active is not real active. On an ASA you can have multiple virtual firewalls called contexts. These are literally separate firewalls on the same physical device. Contexts are useful for multiple customers using the same firewall or perhaps splitting up the firewall for different departments within your company.

If you only have one context then you can only run the firewalls in active/standy. If you want to run the firewalls in active/active then you need at least 2 contexts eg.

context 1 (c1) can be active on asa1 and standby on asa2

context 2 (c2) can be active on asa2 and standby on asa1

but c1 could not be active on both asa1 and asa2.

2) If you have 2 firewalls you are going to need a switch or preferably 2 switches between the firewalls and the routers. These switches are not for DMZs but simply to interconnect your firewalls to your routers. You can use them for DMZs as well using vlans but it is generally not recommended to use the same switches for the outside and the DMZs.

3) Using both ISPs simultaneously is relatively easy for outbound internet access. Inbound to any servers you want accessed from the internet is a bit more tricky as mentioned before. If you have provider independant addressing then you can advertise this space to both ISPs which makes it easier. If you have a block per ISP then it can be a bit more complicated.

The major issue is that if you have 2 ISP blocks and use one of them for these servers then if that ISP goes down you need to remap the private server IPs to the other ISP block. Not only that but obviously the DNS servers on the internet need to know the new IPs. You may be able to advertise ISP1 block out of ISP2 but it is dependant on the ISP.

4) With the ASA you can have multiple default routes pointing out of the same interface (but not multiple default routes out of different interfaces). So instead of GLBP which won't work if you want to use both ISPs you can either -

i) have 2 default-routes on the firewall pointing to both routers. If you do this though as mentioned previously you then need to check the availability of the routers.

ii) run a dynamic routing protocol between the routers and the firewalls. Then yes, each firewall would receive equal cost paths from both routers.

Jon