CSM Bridged mode config issue

Unanswered Question
May 25th, 2010
User Badges:

I currently have a CSM that is load balancing two web servers.  Everything working great.  I have two new web servers that are being used for a different system so I basically copied the old config, changed the names of the vservers, serverfarms and policies and expected the same result as the first.


What is happening is that when I ping the VIP, it gets redirected to one of the reals but then the real responds back instead of the VIP.


Not sure why that is happening.

  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 0 (0 ratings)
Loading.
Sean Merrow Wed, 05/26/2010 - 08:49
User Badges:
  • Silver, 250 points or more

Hello,


It sounds like the server's response to the PING is bypassing the CSM, and therefore the CSM cannot source-NAT the server's IP address back to the VIP before forwarding it to the client.  Make sure that the server's response is going back to the CSM.


If this does not help, then please come back with the config of the CSM and let us know the default gateway of the real server.


Thanks,

Sean

jlhainy Wed, 05/26/2010 - 12:33
User Badges:

short of bringing in a sniffer, is there any other way to veryify that the server's response is going back to the CSM?

Gilles Dufour Thu, 05/27/2010 - 06:39
User Badges:
  • Cisco Employee,

Configure client nat and if the problem goes away you now it was an asymetric problem.



G.

jlhainy Tue, 06/08/2010 - 07:13
User Badges:

Giles,


Are you referring to the client nat in the serverfarm portion of my config?

Sean Merrow Tue, 06/08/2010 - 09:57
User Badges:
  • Silver, 250 points or more

Hello,


That is exactly what Gilles was referring to.  Here is an example:


vlan 20 client
  ip address 20.20.20.2 255.255.255.0
  gateway 20.20.20.1
!
vlan 10 server
  ip address 10.10.10.1 255.255.255.0
!
natpool SRC-NAT 10.10.10.100 10.10.10.100 netmask 255.255.255.0
!
probe PING icmp
!
serverfarm WEB
  nat server
  nat client SRC-NAT
  real 10.10.10.11
   inservice
  real 10.10.10.12
   inservice
  real 10.10.10.13
   inservice
  probe PING
!
vserver WEB
  virtual 20.20.20.100 tcp www
  no persistent rebalance
  serverfarm WEB
  inservice
!   


If the adding source-NAT resolves the issue, then you know that asymmetric routing was your problem.  One solution would be to leave the source-NAT config in permanently.  The other would be to set the default gateway of your new servers to the CSM interface, and another would be to use policy-based routing.


Hope this helps,

Sean

jlhainy Tue, 06/08/2010 - 12:20
User Badges:

Giles and Sean,


I am running this set up in bridged mode so my client vlan and server vlan are actually in the same subnet, even though they are in different vlans.  If I understand it correctly, it is the CSM that bridges the two vlans together via a common IP address and subnet.  So in this case, is client nat still needed?


vlan 603 client
  ip address 192.168.73.235 255.255.255.192
  gateway 192.168.73.193
!
vlan 502 server
  ip address 192.168.73.235 255.255.255.192

!
serverfarm EXCHANGEBR
  nat server
  no nat client
  predictor leastconns
  real 192.168.73.250
   inservice

!
vserver EXCHANGE
  virtual 192.168.73.252 any
  serverfarm EXCHANGEBR
  sticky 480 group 7
  reverse-sticky 7
  persistent rebalance
  slb-policy EXCHANGE
  inservice

Sean Merrow Tue, 06/08/2010 - 14:09
User Badges:
  • Silver, 250 points or more

Generally no.  You typically wouldn't need source-NAT in a bridge mode configuration.  However, it was recommended as a way to

see if your problem was because of asymmetric routing.  The point of all this is that the server's response to a load balanced connection MUST come back to the CSM for processing before it goes back to the client.  Typcially, the default gateway of the real server will be a router/MSFC or firewall upstream of the CSM on the client VLAN.


Does this help?


Sean

jlhainy Tue, 06/08/2010 - 14:48
User Badges:

Ah!  That make sense.  So, what it looks like is that connections are comming in via the VIP then being passed to a real sever, but

the real server is responding via a different gateway directly instead of going back through the CSM and the VIP.  The gateway of my server is the same gateway that is listed in the client vlan in the config above.  Also, I would like to understand the natpool command a little bit better.

Sean Merrow Wed, 06/09/2010 - 06:57
User Badges:
  • Silver, 250 points or more

Hello,


Typically, the rservers would use the same gateway you have configured on the client VLAN.  The important thing to make sure of, is that you must make sure that the ONLY for these rservers to reach their gateway is through the CSM that is bridging the servers' VLAN to that client VLAN.


As for the natpool command:


natpool SRC-NAT 10.10.10.100 10.10.10.100  netmask 255.255.255.0
!
serverfarm WEB
  nat server
  nat client SRC-NAT


This configuration will tell the CSM to source NAT any client IP address to an IP address in the specified NAT pool.  As you can see in my example, I only have one IP address in the pool.  I can get away with this because the CSM also does PAT (port address translation) by default.  If you've ever seen an ACE config, you need to add the pat keyword at the end of the natpool definition to enable PAT.


With this configuration, the rserver that receives the connection will see the connection as coming from 10.10.10.100, which is owned by the CSM.  Therefore, it will send its response to 10.10.10.100.  When the CSM gets it, it will perform destination NAT and change the destination from 10.10.10.100 back to the real client address.  This is in addition to changing the source address from the real server's address back to the VIP before sending to client.  It sounds like it is this last step that is not happening for you.


HTH,

Sean

jlhainy Mon, 06/14/2010 - 14:16
User Badges:

Sean,


When you said "Typically, the rservers would use the same gateway you have configured on the client VLAN.  The important thing to make sure of, is that you must make sure that the ONLY for these rservers to reach their gateway is through the CSM that is bridging the servers' VLAN to that client VLAN."


Now I assume you meant to say "Typically, the rservers would use the same gateway you have configured on the client VLAN.  The important thing to make sure of, is that you must make sure that the ONLY way for these rservers to reach their gateway is through the CSM that is bridging the servers' VLAN to that client VLAN.


Well, I have a working bridging configuration for a different system and I have found that the real servers in my server vlan do have the client vlan IP address... But the server vlan is in fact a layer 2 vlan, it does not have it's own gateway so it has no other way out other than through the CSM and to the client vlan gateway, just as you said.


What I have found is that the server vlan for my new set up actually has its own gateway.  Because of other servers in this vlan I cannot do away with it.  So, I looked at an ealier post where you stated" If the adding source-NAT resolves the issue, then you know that asymmetric routing was your problem.  One solution would be to leave the source-NAT config in permanently.  The other would be to set the default gateway of your new servers to the CSM interface, and another would be to use policy-based routing."


The two solutions I am interested in is the client nat and the setting of the default gateway of the new servers to the CSM interface.  Exaclty what interface are you referring?  Are you referring to the IP address that bridges the client and server vlan together?


Regarding your client nat example, you mentioned that the client nat address is owned by the CSM, but in your example config I did not see that IP address at all so I am a little confused as to how the csm owns this IP.


I really appreciate your responses!

Sean Merrow Tue, 06/15/2010 - 05:25
User Badges:
  • Silver, 250 points or more

Hello,


You are correct, Ieaving out the word way was a typo.  Sorry for the confusion.


My suspicion at this point is that your problem lies in your following statement:


Well, I have a working bridging configuration for a different system and I have found that the real servers in my server vlan do have the client vlan IP address... But the server vlan is in fact a layer 2 vlan, it does not have it's own gateway so it has no other way out other than through the CSM and to the client vlan gateway, just as you said.


It is very possible that your server is sending its response to the client out its client VLAN interface, which would cause it to bypass the CSM.  This is your symptom that we are chasing down, so this must be investigated.


Using the CSM interface as the default gateway of the servers is not recommended for a bridge-mode configuration.  When I stated that, I had forgotten that we were talking about bridge-mode.  This is common for routed-mode configurations.


When you use source NAT, I stated that the CSM 'owns' the IP address because the IP address is configured in the NAT pool on the CSM.  Nobody else on the subnet should be using an address that is configured in the CSM's NAT pool.


Hope this helps,

Sean

jlhainy Tue, 06/15/2010 - 14:13
User Badges:

ok, it looks like the client nat is really the only answer for my situation.  If I have a VIP that is in the client vlan say 192.168.72.250 and it is mapped to the serverfarm that has real servers of 192.168.72.251 and 252.  This entire vlan is a /26 and the IP that bridges the client and server vlans together is 192.168.73.235.  Does my client src nat pool need to be an IP address that is in this same subnet and does it matter what it is?

jlhainy Mon, 06/28/2010 - 11:27
User Badges:

Sean,


Does my last question/post make sense?

Sean Merrow Mon, 06/28/2010 - 11:47
User Badges:
  • Silver, 250 points or more

Sorry, must have missed the e-mail with your last response.  Yes, I would recommend using one or more available IP addresses in the same subnet

for your nat-pool.


Sean

Actions

This Discussion