Buy or Renew
Buy or Renew
Cisco + Splunk: It’s a new day for your data. Learn more.
 
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
cancel
Start a conversation
2304
Views
15
Helpful
3
Replies

ACE Load Balancer not balancing

Go to solution
jason.vongrabe
Level 1
Level 1

‎08-30-2010 01:57 AM

Hi,

I'm having problems with our ACE Loadbalancer. It doesn't seem to recover after having restarted the Application the last time. I can't rule out having changed something on the Applcation Server, as we had installed a newer Version. I can see that the "gif" is beeing successfully probed. I've trace what is going on, just can't see if I'm missing something. We've always had a problem after starting the Applcation Servers that it takes foreever untill they start getting balanced again.

Can someone look at the Config and tell me if they see a mistake in it?

I have three instances accesst, accesst2 and accesst3. Each instance has 4 Oracle Application Server Containers Deployed on 2 different Apllication Servers. The Site is split between 2 DMZs which are seperated by a Firewall. The Cisco Ace has one leg in each vlan (191 and 195). We always had a problem after taking the Applcation Servers down Updates that it takes forever untill the ACE Server starts blancing agian. For the last 4 Days it hasn't started reblancing yet. As far as I know nothing has changed in the Configuration of the Server (routing or Firewall) or of the ACE. The Firewall Admin said he tried to find the problem, but didn't change anything.

I used ethereal to trace everything and it looks like the ssl Proxy forwards the request to the ACE The Ace passes it on to one of the containers on the Application server and the application server tries to give it back directly to the ssl server and fails. It should be going through the ACE loadbalancer to pass it back due to a url rewrite. Neither the ACE nor the SSL gets a answer.

Do I maybe have a mistake in the ACE Config? Am I missing something here?

######################################################

MS4_ACE_PU/MY-APP# sh running-config
Generating configuration....


logging buffered 7

access-list anyone line 8 extended permit ip any any


probe http HEAD_1
  port 7791
  interval 10
  faildetect 15
  passdetect interval 15
  receive 2
  request method head url /APPLICATION/images/probe.gif
  expect status 200 200
  open 2
probe http HEAD_2
  port 7792
  interval 5
  faildetect 15
  passdetect interval 15
  receive 2
  request method head url /APPLICATION/images/probe.gif
  expect status 200 200
  open 2
probe http HEAD_3
  port 7793
  interval 5
  faildetect 15
  passdetect interval 15
  receive 2
  request method head url /APPLICATION/images/probe.gif
  expect status 200 200
  open 2
probe http HEAD_4
  port 7794
  interval 5
  faildetect 15
  passdetect interval 15
  receive 2
  request method head url /APPLICATION/images/probe.gif
  expect status 200 200
  open 2
probe http HEAD_5
  port 7795
  interval 5
  faildetect 15
  passdetect interval 15
  receive 2
  request method head url /APPLICATION/images/probe.gif
  expect status 200 200
  open 2
probe http HEAD_6
  port 7796
  interval 5
  faildetect 15
  passdetect interval 15
  receive 2
  request method head url /APPLICATION/images/probe.gif
  expect status 200 200
  open 2
probe http HEAD_7
  port 7797
  interval 5
  faildetect 15
  passdetect interval 15
  receive 2
  request method head url /APPLICATION/images/probe.gif
  expect status 200 200
  open 2
probe http HEAD_8
  port 7798
  interval 5
  faildetect 15
  passdetect interval 15
  receive 2
  request method head url /APPLICATION/images/probe.gif
  expect status 200 200
  open 2


parameter-map type http PERSIST-REBALANCE
  persistence-rebalance

action-list type modify http LOCATION-RW-VIP-2
  header rewrite response location header-value "http://accesst3.my-site.de:.....(.*)" replace "https://accesst3.my-site.de/%1"
  header rewrite response content-lokation header-value "http://accesst3.my-site.de:.....(.*)" replace "https://accesst3.my-site.de/%1"
action-list type modify http LOCATION-RW-VIP-1
  header rewrite response content-lokation header-value "http://accesst2.my-site.de:.....(.*)" replace "https://accesst2.my-site.de/%1"
  header rewrite response location header-value "http://accesst2.my-site.de:.....(.*)" replace "https://accesst2.my-site.de/%1"
action-list type modify http LOCATION-RW-VIP
  header rewrite response location header-value "http://accesst.my-site.de:.....(.*)" replace "https://accesst.my-site.de/%1"
  header rewrite response content-lokation header-value "http://accesst.my-site.de:.....(.*)" replace "https://accesst.my-site.de/%1"

rserver host server103
  description KS ApplicationServer
  ip address 10.200.105.33
  inservice
rserver host server104
  description KS ApplicationServer
  ip address 10.200.105.34
  inservice

serverfarm host HTTP-APPL
  rserver server103 7791
      probe HEAD_1
    inservice
  rserver server103 7792
    probe HEAD_2
    inservice
  rserver server104 7791
    probe HEAD_1
    inservice
  rserver server104 7792
    probe HEAD_2
    inservice
serverfarm host HTTP-APPL-1
  rserver server103 7795
    probe HEAD_5
    inservice
  rserver server103 7796
    probe HEAD_6
    inservice
  rserver server104 7795
    probe HEAD_5
    inservice
  rserver server104 7796
    probe HEAD_6
    inservice
serverfarm host HTTP-APPL-2
  rserver server103 7797
    probe HEAD_7
    inservice
  rserver server103 7798
    probe HEAD_8
    inservice
  rserver server104 7797
    probe HEAD_7
    inservice
  rserver server104 7798
    probe HEAD_8
    inservice

sticky http-header TranSON_Cert_Subject group1
  replicate sticky
  serverfarm HTTP-APPL
sticky http-header TranSON_Cert_Subject group2
  replicate sticky
  serverfarm HTTP-APPL-1
sticky http-header TranSON_Cert_Subject group3
  replicate sticky
  serverfarm HTTP-APPL-2

class-map type http inspect match-any HTTP-INS-VIP
  2 match header Host header-value "accesst.my-site.de"
class-map type http inspect match-any HTTP-INS-VIP-1
  2 match header Host header-value "accesst2.my-site.de"
class-map type http inspect match-any HTTP-INS-VIP-2
  2 match header Host header-value "accesst3.my-site.de"
class-map match-all HTTP-VIP
  2 match virtual-address 10.200.101.64 tcp eq www
  class-map match-all HTTP-VIP-1
  2 match virtual-address 10.200.101.68 tcp eq www
class-map match-all HTTP-VIP-2
  2 match virtual-address 10.200.101.69 tcp eq www

policy-map type loadbalance first-match HTTP-SF
  class class-default
    sticky-serverfarm group1
    action LOCATION-RW-VIP
policy-map type loadbalance first-match HTTP-SF-1
  class class-default
    sticky-serverfarm group2
    action LOCATION-RW-VIP-1
policy-map type loadbalance first-match HTTP-SF-2
  class class-default
    sticky-serverfarm group3
    action LOCATION-RW-VIP-2

policy-map type inspect http all-match INS-PM-VIP
  class HTTP-INS-VIP
    permit
policy-map type inspect http all-match INS-PM-VIP-1
  class HTTP-INS-VIP-1
    permit
policy-map type inspect http all-match INS-PM-VIP-2
  class HTTP-INS-VIP-2
    permit

policy-map multi-match SLB-logic
  class HTTP-VIP
    loadbalance vip inservice
    loadbalance policy HTTP-SF
    loadbalance vip icmp-reply active
    loadbalance vip advertise active
    appl-parameter http advanced-options PERSIST-REBALANCE
  class HTTP-VIP-1
    loadbalance vip inservice
    loadbalance policy HTTP-SF-1
    loadbalance vip icmp-reply active
    loadbalance vip advertise active
    appl-parameter http advanced-options PERSIST-REBALANCE
  class HTTP-VIP-2
    loadbalance vip inservice
    loadbalance policy HTTP-SF-2
    loadbalance vip icmp-reply active
    loadbalance vip advertise active
    appl-parameter http advanced-options PERSIST-REBALANCE

interface vlan 191
  ip address 10.200.101.65 255.255.255.0
  alias 10.200.101.67 255.255.255.0
  peer ip address 10.200.101.66 255.255.255.0
  access-group input anyone
  service-policy input SLB-logic
  no shutdown
interface vlan 195
  ip address 10.200.105.65 255.255.255.0
  alias 10.200.105.63 255.255.255.0
  peer ip address 10.200.105.66 255.255.255.0
  access-group input anyone
  no shutdown

#####################################################

Destination         Gateway          Interface         Flags
------------------------------------------------------------------------
10.200.101.0/24     0.0.0.0          vlan191           IA [0x30]
10.200.105.0/24     0.0.0.0          vlan195           IA [0x30]

#####################################################

This is what I get from sh conn detail:

GSKV_ACE_PU/MY-APP# sh conn detail

total current connections : 4

conn-id    np dir proto vlan source                destination           state
----------+--+---+-----+----+---------------------+---------------------+------+
4090142    1  in  TCP   191  10.200.101.73:41135   10.200.101.64:80      ESTAB
          [ idle time   : 00:00:21,   byte count  : 835        ]
          [ elapsed time: 00:00:21,   packet count: 3          ]
4090143    1  out TCP   195  10.200.105.33:7791    10.200.101.73:1106    INIT
          [ conn in reuse pool : FALSE]
          [ idle time   : 00:00:21,   byte count  : 0          ]
          [ elapsed time: 00:00:21,   packet count: 0          ]
4090158    1  in  TCP   191  10.200.101.73:41136   10.200.101.64:80      ESTAB
          [ idle time   : 00:00:19,   byte count  : 888        ]
          [ elapsed time: 00:00:19,   packet count: 3          ]
4090159    1  out TCP   195  10.200.105.33:7791    10.200.101.73:1108    INIT
          [ conn in reuse pool : FALSE]
          [ idle time   : 00:00:19,   byte count  : 0          ]
          [ elapsed time: 00:00:19,   packet count: 0          ]

###########################################################

And sh probe (two Conntainers are disabled because I am trying to land on only one Server to find out what is going on.):

GSKV_ACE_PU/MY-APP# sh probe

probe       : HEAD_1
type        : HTTP 
state       : ACTIVE
----------------------------------------------
   port      : 7791    address     : 0.0.0.0         addr type  : -          
   interval  : 10      pass intvl  : 15              pass count : 3          
   fail count: 15      recv timeout: 2                                       
                       --------------------- probe results --------------------
   probe association   probed-address  probes     failed     passed     health
   ------------------- ---------------+----------+----------+----------+-------
   real        : mapp103[7791]                                                
     serverfarm: HTTP-APPL                                                     
                       10.200.105.33   39197      23318      15879      SUCCESS
   real        : mapp104[7791]                                                 
     serverfarm: HTTP-APPL                                                     
                       10.200.105.34   0          0          0          DISABLED

probe       : HEAD_2
type        : HTTP 
state       : ACTIVE
----------------------------------------------
   port      : 7792    address     : 0.0.0.0         addr type  : -          
   interval  : 5       pass intvl  : 15              pass count : 3          
   fail count: 15      recv timeout: 2                                       
                       --------------------- probe results --------------------
   probe association   probed-address  probes     failed     passed     health
   ------------------- ---------------+----------+----------+----------+-------
   real        : mapp103[7792]                                                
     serverfarm: HTTP-APPL
                       10.200.105.33   55090      23333      31757      SUCCESS
   real        : mapp104[7792]                                                 
     serverfarm: HTTP-APPL                                                     
                       10.200.105.34   0          0          0          DISABLED

###################################################################

GSKV_ACE_PU/MY-APP# sh service-policy

Policy-map : SLB-logic
Status     : ACTIVE
-----------------------------------------
Interface: vlan 191
  service-policy: SLB-logic
    class: HTTP-VIP
      loadbalance:
        L7 loadbalance policy: HTTP-SF
        Regex dnld status    : SUCCESSFUL
        VIP Route Metric     : 77
        VIP Route Advertise  : ENABLED-WHEN-ACTIVE
        VIP ICMP Reply       : ENABLED-WHEN-ACTIVE
        VIP State: INSERVICE
        curr conns       : 1         , hit count        : 53
        dropped conns    : 52
        client pkt count : 188       , client byte count: 47195
        server pkt count : 0         , server byte count: 0
        conn-rate-limit      : 0         , drop-count : 0
        bandwidth-rate-limit : 0         , drop-count : 0
        Parameter-map(s):
          PERSIST-REBALANCE

Labels:
0 Helpful
1 Accepted Solution

Accepted Solutions

Go to solution
litrenta
Level 3
Level 3
In response to jason.vongrabe

‎08-30-2010 08:36 AM

Yes you need the nat, If the server uses the firewall as the default gateway then the return traffic to the client will not come back to the ace unless you nat to an address the ace owns. so this is correct .

View solution in original post

3 Replies 3

Go to solution
litrenta
Level 3
Level 3

‎08-30-2010 08:18 AM

does the server 10.200.105.33 point to 10.200.105.63as its default gateway ?

we see connection to server in INIT state meaning we have not seen a syn-ack back after sending a syn.

Are the servers dual nic ? IF so is nic teaming set to loadbalance or fault tolerant

? if its set to loadbalance we will get failures because the return from the sever could

be from a different mac source than what the ace had as a destination mac when it sent the re

quest . Ace will drop this as a mac spoofing violation. IF nic teaming you always want to

be set for fault tolerant on the server.

Go to solution
jason.vongrabe
Level 1
Level 1
In response to litrenta

‎08-30-2010 08:25 AM

Hi,

the Server has a default Gateway pointing to the Firewall. I just changed my Configuration on the ACE so it uses nating like this:

policy-map multi-match SLB-logic
  class HTTP-VIP
    loadbalance vip inservice
    loadbalance policy HTTP-SF
    loadbalance vip icmp-reply active
    loadbalance vip advertise active
    nat dynamic 100 vlan 195
    appl-parameter http advanced-options PERSIST-REBALANCE
  class HTTP-VIP-1
    loadbalance vip inservice
    loadbalance policy HTTP-SF-1
    loadbalance vip icmp-reply active
    loadbalance vip advertise active
    nat dynamic 100 vlan 195
    appl-parameter http advanced-options PERSIST-REBALANCE
  class HTTP-VIP-2
    loadbalance vip inservice
    loadbalance policy HTTP-SF-2
    loadbalance vip icmp-reply active
    loadbalance vip advertise active
    nat dynamic 100 vlan 195
    appl-parameter http advanced-options PERSIST-REBALANCE

interface vlan 191
  ip address 10.200.101.66 255.255.255.0
  alias 10.200.101.67 255.255.255.0
  peer ip address 10.200.101.65 255.255.255.0
  access-group input anyone
  service-policy input SLB-logic
  no shutdown
interface vlan 195
  ip address 10.200.105.66 255.255.255.0
  alias 10.200.105.63 255.255.255.0
  peer ip address 10.200.105.65 255.255.255.0
  access-group input anyone
  nat-pool 100 10.200.105.200 10.200.105.200 netmask 255.255.255.255 pat
  no shutdown

It seems to work now. I haven't tested it with any load yet though. It would be nice if you tell me if it's set right like this (never did it with natting before).

Thanks for your Help and your answer! I appreciate it.

Jason

Go to solution
litrenta
Level 3
Level 3
In response to jason.vongrabe

‎08-30-2010 08:36 AM

Yes you need the nat, If the server uses the firewall as the default gateway then the return traffic to the client will not come back to the ace unless you nat to an address the ace owns. so this is correct .

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community:

Customers Also Viewed These Support Documents