Okay, no problem. The easiest way as far as i am concerned is to route with the FWSM and bridge with the CSM. Note that you put CSS but the CSM is the 6500 module and the principles are the same. 

So switches with FWSM and CSM we'll call distro1 and distro2. 

On these switches you would have a vlan that is for communicating between the MSFC and the outside interface of the FWSM. This is the only vlan that the FWSM knows about that also has an SVI on the 6500. All other vlans that are assigned to the FWSM only exist at layer 2 on the 6500 switches and their L3 interfaces are FWSM interfaces. 

I’m sorry, but I am a bit confused by this. Can you show me how you would implement such a set up? It may just be the words you’re using – and the fact that I’m stupid.  

No it’s probably me. All these things are easier with diagrams. I might still have some at work so I’ll see if I can dig them out. 

What I’m trying to explain is that if you want the FWSM to firewall a vlan you cannot have an SVI on the MSFC for that vlan otherwise traffic will just get routed around the firewall. An example 

Vlan 5   - vlan to connect MSFC to FWSM 192.168.5.0/29
vlan 10  - server vlan 1  192.168.6.0/24
vlan 20  - server vlan 2  192.168.7.0/24

You want to firewall vlan 20 but not vlan 10. In this example we won’t worry about load-balancing, it keeps things simpler.

1)	You need to create all 3 vlans at L2 so if you did a “sh run” on the 6500 you would see all 3 vlans. 
2)	You then create an SVI (interface vlan 10) on the MSFC for the servers on vlan 10 as you are not firewalling these servers.
3)	You then create an SVI for vlan 5 on the MSFC.  – 192.168.5.1 
4)	You then log onto the FWSM (it’s config is stored separately on the FWSM module) and create an outside interface in vlan 5. – 192.168.5.2
5)	You then also create a dmz interface on the FWSM for vlan 20. You do not create an SVI on the 6500 for this vlan.
6)	You then add a route to the MSFC – ip route 192.168.7.0 255.255.255.0 192.168.5.2, which says to get to the servers in vlan 20 go to the outside interface of the FWSM.

Note that the FWSM in single context mode will support OSPF if you don’t like the idea of all the statics.

So a client want to talk to a server in vlan 10. The packet gets routed to the 6500 and then gets routed out onto vlan 10 via the MSFC. 

A client wants to talk to a server in vlan 20. The packet gets routed to the 6500, the 6500 sees the static route and routes the packet out of it’s vlan 5 interface to the 192.168.5.2 address – ie the outside interface of the FWSM. The FWSM then forwards the packet out onto vlan 20. The server responds by sending the return packet back to it’s default-gateway which is the vlan 20 interface on the FWSM. 

So a “sh vlan” on the 6500 would show vlan 5, 10 & 20.
A “sh ip int br” would show interface vlans for 5 & 10 but not 20. 

To run the CSM in bridge mode you have 2 vlans but only use 1 ip subnet for both vlans. 

Huh?

Yes, this confused me the first time I came across it. Basically the CSM is acting as a bridge joining 2 vlans together. You only want 1 IP subnet but you have to use 2 vlans because if you used the same vlan on either side of the CSM ( and remember the CSM is acting as a L2 device here ) this would create a L2 loop and STP would have to block one of the ports.

So you end up with 

MSFC -> FWSM -> CSM -> servers

For arguments sake you allocate vlan 10 and vlan 11 as the 2 vlans used for bridging the CSM and vlan 5 for the MSFC to FWSM vlan so it looks like 

MSFC - vlan 5 - FWSM - vlan 10 - CSM - vlan 11 - servers

Addressing wise 

MSFC (192.168.5.1/29) - (192.168.5.2/29) FWSM (192.168.6.1/25 - CSM - (192.168.6.10 - 15 servers. 

Not the best diagram but it should get the key points across. Note that all the VIP's (Virtual IP addresses) for the loadbalancing come from the same subnet ie. 192.168.6.1/25

OK, so there is a VLAN and a routed connection between the MSFC and the FWSM’s OUTSIDE interface, right?  - YES

And the FWSM’s INSIDE interface has a routed connection with the CSM’s <fill in the blank> interface?  - 

Not quite. The connection between the FWSM and the CSM in bridged mode is just layer 2. The CSM does not act as layer 3 device in this scenario so the servers see the FWSM as the next L3 device ie. their default-gateway even though they have to go through the CSM to get to the FWSM. They just don’t know they have to go through the CSM. 

The default-gateway of your servers is the FWSM interface address of 192.168.6.1. The servers are in effect unaware of the CSM but for all traffic to go to and from the FWSM they must go through the CSM. 

This is quite a clean solution and i was told by some of the Cisco engineers at the time that this was the most common deployment type. 

Major drawbacks are 

1) You should make sure that you only have servers you want to load balance in the vlan - vlan 11 in our example. So lets say you have an existing server vlan with 50 servers in it which you want to firewall  but you only want to load balance 10 of them. The above is not such good solution because all traffic to all servers has to go through the CSM which can give performance problems - the CSM is not fabric enabled as such. 

2) Traffic flows. If you are doing "simple" load balancing in the sense of just web load balancing etc. then the above is a good solution. Where it gets tricky is if you are load balancing for a 3 tier app such as ERP. 

With ERP you have clients accessing mid-tiers using a web browser and the mid-tiers then talk to backend database servers. Throw in some single sign on servers etc. into the mix and suddenly you find that there are multiple/complex traffic flows going on ie. 

A client gets load-balanced to a mid-tier. The mid-tier now needs to authenticate the client so it needs to talk to some sign on servers which you also want to load-balance to, so now you have load-balanced servers initating connections to load-balanced servers and it gets messy when all traffic has to keep going back and forth through the CSM. Actually not just messy, there were certain parts of ERP that just didn't work. If you have this type of load balancing then the solution may well be "one-armed" mode. 

So, you’re saying that the BEST way to configure the CSM is in one-armed mode? Is that what you use?

No I’m not saying that one-armed mode is the best. For a start I haven’t deployed the CSM in routed mode in a production environment although I have played around with it in a lab. Others might recommend routed mode. 

What I’m saying is that I have found bridged mode to be very easy to setup and maintain and one-armed mode is more complicated. But it all depends on the traffic flows of your applications.
We didn’t really cover it properly but one-armed mode works as follows. 

In one-armed mode the VIP’s are on a different subnet than the server IP address unlike bridged mode. You also need 2 DMZ’s for one-armed mode, whereas with bridged you only need one. In addition there is the added complication of making sure that traffic gets routed back to the CSM. This is achieved through either PBR or NAT. We will use NAT in our example.

Traffic flow as follows

1)	A client sends a packet to one of the VIPs. This get routed to the 6500 and then the MSFC routes this to the outside interface of the FWSM. 
2)	The FWSM then sends it out onto the VIP dmz to the CSM. 
3)	CSM selects a real server, NAT’s the source IP address to an address that will get routed back to the CSM and then sends it back to the FWSM.
4)	The FWSM then sends the packet out into the server dmz.
5)	The servers responds and sends the packet back to it’s default-gateway ie. the FWSM server vlan interface. The FWSM then sends the packet back out onto the VIP dmz because the destination IP address is routed back to the FWSM. 
6)	The CSM then NAT’s the destination back to the client IP address and sends it back to the FWSM.
7)	FWSM then sends packet back out of outside interface to MSFC.
8)	Packet is routed back to client.

As you can see this is a lot more complicated than bridged mode and traffic goes through the FWSM a few more times.

So it’s a trade off and a lot depends on the applications you are load-balancing.

One-armed mode is where the CSM does not actually sit in the traffic path. So the client connects to a VIP which goes to the distro switches,

Confused again! LOL

…is routed to outside of the FWSM and then the FWSM sends it to the CSM. In this setup the VIP's are not part of the same subnet as the server IP addresses.

When it gets to the CSM it is again load balanced but you NAT the source IP addresses so they appear to come from the CSM itself. This way the return packets then get sent back to the CSM. 

The advantage of this is that to get to the servers direct ie. not load balanced you do not have to go through the CSM at all. So only traffic that needs to be load balanced goes through the CSM. 

This can be a major plus in terms of performance epsecially with some of the more complicated applications. 

Those are the main 2 i have used. Obviously there are other combinations where the FWSM could run in transparent mode and we haven't even touched on contexts and the ability toline up your contexts vertically with the new ACE module so you could allocate a dedicatde context from the FWSM and the ACE module specifically for ERP. Indeed that is what we are doing at the moment in our new data centres. 

I think that's enough to be going on with. Hope some of this has helped. Please come back if you need to.

Jon