Load balancing Questions

Answered Question
May 18th, 2008

Hi, folks:

I have a series of questions to ask consecutively.

Understanding the general idea of what load balancers do is easy enough. It's the actual implementation and how they fit into an existing architecture that I seem to find somewhat challenging.

I don't have access to a school or any formal training, so I am trying to understand the nuts and bolts of implementation on my own.

May I ask for some help?

Please take a look at this link.

http://www.oreillynet.com/images/networking/bourke_1100_image3_big.gif

Here is how I interpret the traffic flow.

Incoming-to-outgoing traffic takes this route:

--> In from Internet --> 208.20.20.200 --> 208.20.20.100 --> 208.20.20.10 --> 208.20.20.1 --> back out to Internet

1a) Am I correct?

2a) Is this considered to be a set up in which the LB is operating in routed mode?

3a) In this case, there would have to be layer 2 adjacency between the server farm access layer and the routed distribution layer, correct?

I'll leave it at that for now.

Thanks

Victor

I have this problem too.
0 votes
Correct Answer by Jon Marshall about 8 years 6 months ago

Your'e going to hate me for this but one-armed mode is not bridged.

What i probably should have said is that in one-armed the traffic goes in and out the same interface hence the one-armed bit.

So traffic from the client hits the MSFC, gets routed to the CSM, the CSM nats the destination to a real server and the source to an address that will get routed back to the CSM and then sends the packet back out of the SAME interface.

As for a book the only one i know is

"Designing Content Switching Solutions" by Zeeshan Naseh and Haroon Khan. It is a cisco press book and it is fairly good. It covers both standalone CSS's and CSM's and also how to integrate these with FWSM's etc. in your data centre. It does not cover the ACE module which is the replacement for the CSM.

Might be an updated version now, worth checking.

I'll have a check this week for docs if i get the chance.

Jon

  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 5 (5 ratings)
Loading.
Jon Marshall Sun, 05/18/2008 - 17:45

Victor

Your'e not the only one who is a bit confused. I'm wondering if some of the addressing is a bit amiss because if you look at the diagram it says the Public Network is 200.20.20.0/24 and not 208.20.20.0/24.

If that is a typo and all the addressing is 208.20.20.x then this cannot be routed mode as you have the same network on the outside and inside of the LB.

So it would be in briged mode where the LB joins together 2 vlans with the same IP subnet in both vlans. But the digram suggests that the LB's are not inline ie. the traffic from the provider routers at the top can go straight to the servers without going to the LB's.

I suspect there could be a typo but to be sure i would post this in the "Application Networking" forum where Gilles (Cisco LB expert) hangs out.

Jon

lamav Sun, 05/18/2008 - 18:38

"So it would be in briged mode where the LB joins together 2 vlans with the same IP subnet in both vlans."

Jon, what does that statement mean? I find it confusing, but I keep seeing it in explanations I read.

Two vlans with the same IP subnet in both vlans?

I really dont get the differentce between bridged mode and routed mode....

Victor

Jon Marshall Sun, 05/18/2008 - 22:43

Okay, there are 3 ways of load balancing

1) Bridged mode

2) Routed mode

3) One-armed mode

For bridged and routed imagine we have a LB that has an outside interface and an inside interface. The inside interface is where the real servers are and the outside interface is where requests to VIP's come in. Also we will do this as if we had a 6500 with CSM's - principle is the same with standalone kit.

Routed mode

===========

With routed mode the VIP's are from a different IP subnet than the real servers. So the outside interface of the LB will have an address out of the VIP subnet or another subnet entirely - up to you. The servers will be on a different IP subnet.

The outside interface of the CSM will share a vlan with the 6500 and there will be an interface on the MSFC with an address out of the VIP subnet.

The servers default-gateway will point to the inside interface of the LB. And the LB acts as a router in so far as it receives packets for one of it's VIP's, translates it to one of the real server addresses and then routes it onto the server subnet.

Bridged mode

============

With bridged mode the VIP and the real server IP addresses are out of the same IP subnet.

So there is one IP subnet and you would set this up as follows

MSFC (192.168.5.1/24) -> CSM -> server (192.168.5.10)

Note there would be more than one server obviously.

But there is one problem with scenario. If you only used one vlan you are creating an STP loop because the CSM would be receiving packets on it's outside interface and forwarding them on it's inside interface. If it helps think of a physical LB - you connect it into the same switch on 2 different ports, so in effect you have created a L2 loop.

The way round this is to use 2 separate vlans which breaks the loop because now the outside and interface are in different vlans. You still use one IP subnet across both vlans though, otherswise it wouldn't be routed.

Just for completeness

One-armed mode

==============

This is where the LB is not inline to the traffic and you have to direct traffic to the LB. Also you have to NAT the source IP addresses to force the traffic to go back to the LB.

I can go into a bit more detail on one-armed if you need it but i don't want to overload you :-)

Jon

lamav Mon, 05/19/2008 - 06:41

Jon, thanks for that really informative explanation. Brilliant, as usual.

There is still one thing Im stuck-on-stupid about:

"The way round this is to use 2 separate vlans which breaks the loop because now the outside and interface are in different vlans. You still use one IP subnet across both vlans though, otherswise it wouldn't be routed."

How do you place the same subnet in 2 different vlans?

To me, a VLAN can contain several subnets:

Example:

interface vlan 10

ip address 10.10.80.0 255.255.254.0

Contains 2 subnets

10.10.80.0/24

10.10.81.0/24

But how do you configure 2 separate VLANs with the same subnet? :-(

What are the main advantages and disadvantages of deploying switched and routed modes?

Thanks, again, buddy. You're the best.

Victor

Jon Marshall Mon, 05/19/2008 - 09:11

Victor

Appreciate the comments.

Yes a vlan can contain several subnets but you can also extend a subnet across 2 vlans. In fact this is how you configure a transparent firewall, a bridged load-balancing configuration etc.

Remeber that the CSM is just "bridging" 2 vlans together at Layer 2, it doesn't know and doesn't care that both vlans are using the same L3 IP subnet. It is not seen as a layer 3 hop by end devices so the servers see the next L3 hop as the MSFC on the 6500. They have no idea that to get there they have to go through the CSM. And because it is bridging this is exactly why you can't use just one vlan.

So to use 2 vlans per subnet you need something to bridge between (or join) the vlans and in this case it is a CSM but it could be an FWSM, an ACE module etc.

Main disadvantages/advantages

Well obviously with L2 bridged mode you have to have the L2 adjacent to the 6500 switches, which reminds me of another conversation we had about L3 routed access-layer :-)

I personally think L2 is easier to set up and if you are firewalling in front of your load-balancing then having the firewall in routed mode and the CSM's in bridged mode is quite a common configuration. Or at least was the last time i checked :-)

With L3 routed mode your server subnets don't have to be adjacent to the 6500 switches so you could do your L3 routed access-layer. Because your VIP's are not out of the same IP subnet then there is no direct relationship between VIP's and real addresses ie. you could have 5 VIP's out of the same subnet but they point to real servers each in their own subnet.

It also seems to depend on the kit. Cisco recommended to use CSM in bridge mode when we were deploying a while back although all of our CSS (standalone LB's) were using routed mode.

Just to confuse you even more !!, we ended up using one-armed mode for our ERP applications because of the amount of traffic flows backwards and forwards. It's important to realise that with both routed and bridged if a server behind the load balancer wants to communicate with another server on a different subnet this traffic has to go through the LB. In a basic http load-balancing setup not a problem but it can be.

Jon

lamav Mon, 05/19/2008 - 10:48

Jon:

Let's see if I got this straight.

1. A load balancer can be deployed in 2 modes: either using a switched/bridged mechanism or routed.

2. A load balancer that uses bridged mode has the server farm's VIP and the real server addresses sitting on the same VLAN. So, the forwarding mechanism between VIP and real server is switched. In this case, a switched access layer is necessary to provide L2 adjacency between the real servers and the VIP. In a routed access layer, the server vlans are confined to the access switch and there is L3 isolation between the access and distribution layers, so there is no way that the VIP can be on the same subnet as an access switch VLAN. Concomitantly, distributed data center services should (almost must, really) be deployed in the distribution layer to allow centralized services to be accessed across the data center.

3. In a routed mode CSM deployment, the server farm VIP sits on a separate L3 domain from the real server subnet. So, there is no L2 adjacency requirement between CSM VIP and real servers, and therefore, a routed access layer will not impede in this situation.

A routed mode LB deployment is also called a NAT-SLB implementation, because the client's destination traffic, which is meant for the VIP, is NAT'ed to the real server address. And on the return journey, the real server address is NAT'ed back to the VIP address to make it look to the client that the server's address is the VIP.

4. A one-armed CSM is one in which the LB is NOT inline with the flow of traffic between the client and the server. (I need some help here). The servers will have their default gateway set to the VIP of the LB (correct?)

How am I doing?

Victor

Jon Marshall Mon, 05/19/2008 - 11:01

"How am I doing?"

Not bad at all :-). Couple of things

"A load balancer that uses bridged mode has the server farm's VIP and the real server addresses sitting on the same VLAN."

Not quite. It's a bit pedantic but it helps to clarify the concepts in your head. The server farm's VIP and real addresses sit on the same IP subnet - they will actually be on different vlans.

"A routed mode LB deployment is also called a NAT-SLB implementation, because the client's destination traffic, which is meant for the VIP, is NAT'ed to the real server address"

Yes, but this also happens for bridge mode as well because the VIP still needs Natting to a real server address even though that address is out of the same IP subnet.

What happens with one-armed is that you need to not only NAT the destination but also the source IP addresses. This is because when the packets hit the real server if the source IP addresses were the original IP address then the server would just send them back to it's default-gateway and they would be routed back to the client. BUT they would not have gone back through the LB on the return path.

So the LB in one-armed mode not only nats the VIP to a real address it also nats the source IP addresses so that they get sent back to the LB from the real server. The LB then nats them back to the real client IP address and sends them back to the MSFC to be routed back.

Jon

lamav Mon, 05/19/2008 - 11:32

"Not quite. It's a bit pedantic but it helps to clarify the concepts in your head. The server farm's VIP and real addresses sit on the same IP subnet - they will actually be on different vlans."

Gotcha. You need to span the subnet across 2 vlans to preempt an STP blocking situation.

"What happens with one-armed is that you need to not only NAT the destination but also the source IP addresses. This is because when the packets hit the real server if the source IP addresses were the original IP address then the server would just send them back to it's default-gateway and they would be routed back to the client."

OK, but isnt the default gateway on the sevrers set to the LB and not the MSFC HSRP VIP? IOW, the server's return taffic would be forwarded to the LB anyway. No?

Victor

Jon Marshall Mon, 05/19/2008 - 11:46

"OK, but isnt the default gateway on the sevrers set to the LB and not the MSFC HSRP VIP? IOW, the server's return taffic would be forwarded to the LB anyway. No? "

No it isn't because if the default-gateway of the servers was set to the LB then you are back to routed mode.

The key thing about one-armed mode is that you must make the traffic go to the LB rather than the traffic just passing through and back through the LB.

Oh, just noticed, thanks for the ratings - much appreciated :-)

Jon

lamav Mon, 05/19/2008 - 11:53

OK, so one armed mode is a bridged implementation. Got it.

Do you know of a really good document or book to read on implementing SLB? I am having a hard time finding an SRND design doc on Cisco's website that focuses on this. I need something that gives everything, soup to nuts, so I can get a comprehensive understanding.

Lastly, do you remember that one time you sent me an explanation of how you deployed LB on your network and you said that you had documents and diagrams you could sen dme to supplement it? Do you still have them and may I see them?

Thanks, man. I really appreciate your time and energy.

Victor

Correct Answer
Jon Marshall Mon, 05/19/2008 - 12:00

Your'e going to hate me for this but one-armed mode is not bridged.

What i probably should have said is that in one-armed the traffic goes in and out the same interface hence the one-armed bit.

So traffic from the client hits the MSFC, gets routed to the CSM, the CSM nats the destination to a real server and the source to an address that will get routed back to the CSM and then sends the packet back out of the SAME interface.

As for a book the only one i know is

"Designing Content Switching Solutions" by Zeeshan Naseh and Haroon Khan. It is a cisco press book and it is fairly good. It covers both standalone CSS's and CSM's and also how to integrate these with FWSM's etc. in your data centre. It does not cover the ACE module which is the replacement for the CSM.

Might be an updated version now, worth checking.

I'll have a check this week for docs if i get the chance.

Jon

lamav Mon, 05/19/2008 - 12:10

Thanks...

By the way, I met Zeehsan a few months back. Cisco and my company were working on a reference architecture doc for a mutual client of ours.

Victor

Jon Marshall Mon, 05/19/2008 - 12:19

Impressive, you move in much higher circles than i do :-)

If you still have his contact details he should be able to tell you what the most common deployments are in practice. Like i say with the service modules routed firewall/bridged load-balancing was Cisco's recommendation when we virtualised our data centre.

When you find out let me know :-)

Jon

Actions

This Discussion