ACE Problem after restarting Application

[jason.vongrabe]

Hi,

we have an ACE20 and have set it up to balance 4 Containers on a Oracle Application Server. Every time we stop all Containers at the same time for longer than an hour it takes forever (hours) until the Load Balancer starts balancing the Containers again. I can see that the ACE Module is checking the Containers in the Apache Logfiles on the Application Server and gets a 200, but still we can't access the Application for a few hours. If I connect direct to the Container it also works fine... just the ACE does not work. Like it has a timeout and is waiting.

Any idea how to give it a kick?

While accessing the Application I can see that it connects, but nothing happens...

sh conn detail

total current connections : 2

conn-id    np dir proto vlan source                destination           state
----------+--+---+-----+----+---------------------+---------------------+------+
68155      2  in  TCP   191  10.200.101.73:35777   10.200.101.64:80      ESTAB
          [ idle time   : 00:00:18,   byte count  : 888        ]
          [ elapsed time: 00:00:18,   packet count: 3          ]
68156      2  out TCP   195  10.200.105.33:80      10.200.101.73:35777   INIT
          [ conn in reuse pool : FALSE]
          [ idle time   : 00:00:18,   byte count  : 0          ]
          [ elapsed time: 00:00:18,   packet count: 0          ]

Thanks for any help!

Jason

[Gilles Dufour · Accepted solution]

From the output you provided, it seems like the server is not responding.

Did you get a sniffer trace to verify what is going ?

Gilles.

[jason.vongrabe]

Hi Gilles,

I haven't sniffered it yet because the server responds if I don't go over the ace. I will check it though.

I was thinking it could have a problem with the firewall. It's wierd that if I leave it over night it normally works in the morning. Just don't understand why. if it would never work, it would make it easier to find the problem.

Jason

[jason.vongrabe]

So I got a chance to trace everything with etherreal, on both the Servers.

It looks like the frontend server (mato) sends the request to the ACE, the ACE forwards the request to the application server (mapp) the package the application server is getting states that it comes from the frontend server (mato). The application server (mapp) tries to answer back to the frontend server (mato) instead of to the ACE Loadbalancer and gets droped by the firewall.

Any idea why the application server answers to the frontend server and not to the ace server? Or am I reading it wrong?

Thanks for any help!

Jason

[jason.vongrabe]

Hi,

I'm still having problems with the ACE Laodbalancer. At the moment it doesn't seem to recover after having restarted the Application the last time.

Can someone look at the Config and tell me if they see a mistake in it?

I have three instances accesst, accesst2 and accesst3. Each instance has 4 Oracle Application Server Containers Deployed on 2 different Apllication Servers. The Site is split between 2 DMZ which are seperated by a Firewall. The Cisco Ace has one leg in each vlan (191 and 195). We always had a problem after taking the Applcation Servers down Updates that it takes forever untill the ACE Server starts blancing agian. For the last 4 Days it hasn't started reblancing yet. As far as I know nothing has changed in the Configuration of the Server or of the ACE. The Firewall Admin said he tried t find a problem, but didn't change anything.

Do I maybe have a mistake in the ACE Config? Am I missing something here?

######################################################

MS4_ACE_PU/MY-APP# sh running-config
Generating configuration....

logging buffered 7

access-list anyone line 8 extended permit ip any any

probe http HEAD_1
  port 7791
  interval 10
  faildetect 15
  passdetect interval 15
  receive 2
  request method head url /APPLICATION/images/probe.gif
  expect status 200 200
  open 2
probe http HEAD_2
  port 7792
  interval 5
  faildetect 15
  passdetect interval 15
  receive 2
  request method head url /APPLICATION/images/probe.gif
  expect status 200 200
  open 2
probe http HEAD_3
  port 7793
  interval 5
  faildetect 15
  passdetect interval 15
  receive 2
  request method head url /APPLICATION/images/probe.gif
  expect status 200 200
  open 2
probe http HEAD_4
  port 7794
  interval 5
  faildetect 15
  passdetect interval 15
  receive 2
  request method head url /APPLICATION/images/probe.gif
  expect status 200 200
  open 2
probe http HEAD_5
  port 7795
  interval 5
  faildetect 15
  passdetect interval 15
  receive 2
  request method head url /APPLICATION/images/probe.gif
  expect status 200 200
  open 2
probe http HEAD_6
  port 7796
  interval 5
  faildetect 15
  passdetect interval 15
  receive 2
  request method head url /APPLICATION/images/probe.gif
  expect status 200 200
  open 2
probe http HEAD_7
  port 7797
  interval 5
  faildetect 15
  passdetect interval 15
  receive 2
  request method head url /APPLICATION/images/probe.gif
  expect status 200 200
  open 2
probe http HEAD_8
  port 7798
  interval 5
  faildetect 15
  passdetect interval 15
  receive 2
  request method head url /APPLICATION/images/probe.gif
  expect status 200 200
  open 2

parameter-map type http PERSIST-REBALANCE
  persistence-rebalance

action-list type modify http LOCATION-RW-VIP-2
  header rewrite response location header-value "http://accesst3.my-site.de:.....(.*)" replace "https://accesst3.my-site.de/%1"
  header rewrite response content-lokation header-value "http://accesst3.my-site.de:.....(.*)" replace "https://accesst3.my-site.de/%1"
action-list type modify http LOCATION-RW-VIP-1
  header rewrite response content-lokation header-value "http://accesst2.my-site.de:.....(.*)" replace "https://accesst2.my-site.de/%1"
  header rewrite response location header-value "http://accesst2.my-site.de:.....(.*)" replace "https://accesst2.my-site.de/%1"
action-list type modify http LOCATION-RW-VIP
  header rewrite response location header-value "http://accesst.my-site.de:.....(.*)" replace "https://accesst.my-site.de/%1"
  header rewrite response content-lokation header-value "http://accesst.my-site.de:.....(.*)" replace "https://accesst.my-site.de/%1"

rserver host server103
  description KS ApplicationServer
  ip address 10.200.105.33
  inservice
rserver host server104
  description KS ApplicationServer
  ip address 10.200.105.34
  inservice

serverfarm host HTTP-APPL
  rserver server103 7791
      probe HEAD_1
    inservice
  rserver server103 7792
    probe HEAD_2
    inservice
  rserver server104 7791
    probe HEAD_1
    inservice
  rserver server104 7792
    probe HEAD_2
    inservice
serverfarm host HTTP-APPL-1
  rserver server103 7795
    probe HEAD_5
    inservice
  rserver server103 7796
    probe HEAD_6
    inservice
  rserver server104 7795
    probe HEAD_5
    inservice
  rserver server104 7796
    probe HEAD_6
    inservice
serverfarm host HTTP-APPL-2
  rserver server103 7797
    probe HEAD_7
    inservice
  rserver server103 7798
    probe HEAD_8
    inservice
  rserver server104 7797
    probe HEAD_7
    inservice
  rserver server104 7798
    probe HEAD_8
    inservice

sticky http-header TranSON_Cert_Subject group1
  replicate sticky
  serverfarm HTTP-APPL
sticky http-header TranSON_Cert_Subject group2
  replicate sticky
  serverfarm HTTP-APPL-1
sticky http-header TranSON_Cert_Subject group3
  replicate sticky
  serverfarm HTTP-APPL-2

class-map type http inspect match-any HTTP-INS-VIP
  2 match header Host header-value "accesst.my-site.de"
class-map type http inspect match-any HTTP-INS-VIP-1
  2 match header Host header-value "accesst2.my-site.de"
class-map type http inspect match-any HTTP-INS-VIP-2
  2 match header Host header-value "accesst3.my-site.de"
class-map match-all HTTP-VIP
  2 match virtual-address 10.200.101.64 tcp eq www
  class-map match-all HTTP-VIP-1
  2 match virtual-address 10.200.101.68 tcp eq www
class-map match-all HTTP-VIP-2
  2 match virtual-address 10.200.101.69 tcp eq www

policy-map type loadbalance first-match HTTP-SF
  class class-default
    sticky-serverfarm group1
    action LOCATION-RW-VIP
policy-map type loadbalance first-match HTTP-SF-1
  class class-default
    sticky-serverfarm group2
    action LOCATION-RW-VIP-1
policy-map type loadbalance first-match HTTP-SF-2
  class class-default
    sticky-serverfarm group3
    action LOCATION-RW-VIP-2

policy-map type inspect http all-match INS-PM-VIP
  class HTTP-INS-VIP
    permit
policy-map type inspect http all-match INS-PM-VIP-1
  class HTTP-INS-VIP-1
    permit
policy-map type inspect http all-match INS-PM-VIP-2
  class HTTP-INS-VIP-2
    permit

policy-map multi-match SLB-logic
  class HTTP-VIP
    loadbalance vip inservice
    loadbalance policy HTTP-SF
    loadbalance vip icmp-reply active
    loadbalance vip advertise active
    appl-parameter http advanced-options PERSIST-REBALANCE
  class HTTP-VIP-1
    loadbalance vip inservice
    loadbalance policy HTTP-SF-1
    loadbalance vip icmp-reply active
    loadbalance vip advertise active
    appl-parameter http advanced-options PERSIST-REBALANCE
  class HTTP-VIP-2
    loadbalance vip inservice
    loadbalance policy HTTP-SF-2
    loadbalance vip icmp-reply active
    loadbalance vip advertise active
    appl-parameter http advanced-options PERSIST-REBALANCE

interface vlan 191
  ip address 10.200.101.65 255.255.255.0
  alias 10.200.101.67 255.255.255.0
  peer ip address 10.200.101.66 255.255.255.0
  access-group input anyone
  service-policy input SLB-logic
  no shutdown
interface vlan 195
  ip address 10.200.105.65 255.255.255.0
  alias 10.200.105.63 255.255.255.0
  peer ip address 10.200.105.66 255.255.255.0
  access-group input anyone
  no shutdown

#####################################################

Destination         Gateway          Interface         Flags
------------------------------------------------------------------------
10.200.101.0/24     0.0.0.0          vlan191           IA [0x30]
10.200.105.0/24     0.0.0.0          vlan195           IA [0x30]