Need help to Configure Cisco ACE 4710 Cluster Deployment

[Pathirage Don Amal Rajapakse]

Dear Experts,

I'm newbie for Cisco ACE 4710, and still I'm in learning stage. Meanwhile I got chance at my work place to deploy a Cisco ACE 4710 cluster which should load balance the traffic between  two Application Servers based on HTTP and HTTPS traffic. So I was looking for good deployment guide in Cisco SBA knowledge base then finall found this guide.

http://www.cisco.com/en/US/docs/solutions/SBA/February2013/Cisco_SBA_DC_AdvancedServer-LoadBalancingDeploymentGuide-Feb2013.pdf

This guide totally fine with my required deployment model. I have same deployment environment as this guide contains with ACE cluster that connects to two Cisco 3750X (Stack) switches. But I have some confusion places in this guide

This guide follow the "One-armed mode" as a deployment method. But when I go through it further I have noticed that they have configured server VLAN as a 10.4.49.0/24 (all servers reside in it) and Client side VIP also in same VLAN which is 10.4.49.100/24 (even NAT pool also).

My confusion is, as I have learned about Cisco ACE 4710 one-armed mode deployment method, it should has two VLAN segments, one for Client side which client request come and hit the VIP and then second one for Server side. which means besically two VLANs. So please be kind enough to go through above document then tell me where is wrong, what shoud I need to do for the best. Please this is an urgent, so need your help quickly.

Thanks....!

-Amal-

[Kanwaljeet Singh]

HI Amal,

The one arm mode doesn't require you to redesign your IP schema as you just place in your load balancer & configure a virtual IP for the traffic to hit and get loadbalanced.

In one-arm mode, you configure the ACE with a single VLAN that handles both client requests and server responses.

Also, it is not necessary to have  separate VLANS for clients and servers. Depends upon requirement and set up.

Ace can function in ROUTED and BRIDGE MODE and in routed mode you can have one VLAN mode set up(one arm) and two vlan mode as well. In two vlan set up ACE will route the traffic from client to server unlike one arm set up where MFSC is doing the same.

Below is from user guide:

For one-arm mode, you must configure the ACE with client-source network address translation (NAT) or policy-based routing (PBR) to send requests through the same VLAN to the server.  In one arm mode the ACE is not inline with the traffic and receives and sends requests through the Multilayer Switching Feature card (MSFC) that acts as a default gateway to the servers. The MSFC routes requests to a VIP that is configured on the ACE. When the ACE selects the server for the request based on the configured policy, it rewrites the source IP address with an address in the NAT pool. Then the ACE forwards the request to the server on the same VLAN through the default gateway on the MSFC.

The server sends a response to the default server gateway on the MSFC. The server response contains its source IP address and the NAT address of the ACE as the destination IP address. The MSFC forwards the response to the ACE. The ACE receives the response, changes the source IP address to the VIP, and sends it to the MFSC. Then the MFSC forwards the response to the client.

Please visit the below link:

http://www.cisco.com/en/US/docs/interfaces_modules/services_modules/ace/vA5_1_0/configuration/getting/started/guide/one_arm.html

Let me know if you have any questions.

Regards,

Kanwal

[Pathirage Don Amal Rajapakse]

Dear Kanwal,

First of all thank you so much for your quick reply. I read the guide you have shared with me above. In there they have clearly mentioned we should have seperate VLAN for servers (which we don't need to change any IPs of servers) and we should have another VLAN which available for both clients and servers. This means we have to have two VLANs that one for ACE and another for servers (Already created). In that guide they have used 192.168.5.0/24 subnet (VLAN) for servers and they have introduced 172.16.5.0/24 as a new subnet (VLAN) for ACE which call "Client and ServerVLAN 100" VLAN that routable to Server VLAN.

In my work place Network I have only two VLANs to utilize, one is Management VLAN 172.25.20.0/24 (For Device Management) and other one for Server VLAN 172.25.45.0/24. My APP Servers reside in this VLAN. But the guide I have shared with you in my first post,

http://www.cisco.com/en/US/docs/solutions/SBA/February2013/Cisco_SBA_DC_AdvancedServer-LoadBalancingDeploymentGuide-Feb2013.pdf

They haven't introduced any seperate VLAN as "Client and ServerVLAN 100" like the guide mentioned that you have shared with me above. What they have done is, they utilized only one VLAN 149 (10.4.49.0/24) which all the real servers reside in it, and the Cisco ACEs IPs too, ACE VIP 10.4.49.100/24, MGT peer 1 & 2 IPs: (10.4.49.119, 10.4.49.120)/24, and finally NAT pool 10.4.49.99. If they are utilizing same VLAN why do they need NAT ? I'm confusion here.

Following are the Pre Requisites I received from my client to configure ACEs.

Following detail required for configuring Oracle EBS Apps tier on HA:

- LBR IP and Name required to configure EBS APPS Tier (i.e, ap1ebs & ap2ebs nodes)

- Suggested IP and Name for LBR:

  IP : 172.25.45.x (My Server VLAN) [should be on same 172.25.45 subnet of ap1ebs & ap2ebs nodes]

- ebiz.xxxx.lk [on port 80 for http protocol accessibility]

- This LBR IP & name must be resolve and respond on DNS network

If I can follow the guide you have shared with me above simply I just only need to introduce new VLAN for ACEs which is routable to server VLAN thorugh my core switch, but in above my client's second point says my ACEs should be reside in same VLAN which servers reside (172.25.45.0/24). which means my ACEs VIP should be (172.25.45.x - Server VLAN) then all the client requests come and hit on that VIP which reside in my Server VLAN. How can I achive it by using "One Armed method" without introduce a seperate new routable VLAN for ACE as your guide mentioned.

My next question is, If I'll able to deploy my ACE cluster successfully by using "One Armed method", Do I able to cater following requiements which I've got from my client side ?

- Client PC must be able to ping/resolve hostname of the virtual server (LBR IP & NAME)

- EBS Apps tiers must be able to ping/resolve hostname of the virtual server (LBR IP & NAME)

- Loop Back requirement for load balancer compulsory, hardware vendor should enable it if not enable by default

- Enable monitor / health check / probe in LBR:

         - In general, there are 4 types of generic health checks that are available on most hardware load balancers:

                   - Node Checks: IP Layer, ping host

                   - Service Checks: TCP Layer, checks http port can be opened

                   - Content Checks: Application layer, GET /HTTP/1.0, GET /OA_HTM/OAInfo.jsp

                   - Interactive Checks: Custom scripts

- Predictive (node) Load Balancing Method:

         - Fastest(node): passes a new connection based on the fastest response of all pools of Servers

- Session persistence / session stickiness required on LBR to maintain the session state on EBS Apps tiers 

- Make sure that the software load balancer is configured in stateful mode. Also,enable session stickiness (session     

  persistence) in the load balancer and set it to "cookie" or "jsessionid"

I'm looking forward to hearing from you 

Thanks.....!

-Amal-

[Kanwaljeet Singh]

Hi Amal,

If you have clients and servers in same subnet then you need src NAT so that returning traffic from server goes back via loadbalancer. If there is no NAT then server can reply directly to client since they are in same subnet. That means ACE will only be able to see one leg of a connection and thus the problem. Client is expecting a reply from ACE VIP but gets reply from server IP. This is going to a problem when working with TCP.

2nd situation where you would need NAT is even if client is not in same subnet as servers are, if "servers default gateway" is not ACE.In that case for returning traffic to go via ACE you would do src NAT.

If clients are on a different subnet and default GW of servers is ACE you don't need NAT.

In your case all the requirements you mentioned are doable on ACE.

Check ACE user/configuration guide for different predictor methods, different probe methods, different persistent methods etc. and see which one's suit your requirement. Client PC should be able to resolve hostname and that  should be taken care at your DNS system. VIP would be pingable after you configure "loadbalance vip icmp-reply" under class in policy multi-match.

The link that i sent showed a different example. Depending your requirment you can go for one vlan mode or two where ACE VIP and servers are in different VLAN.

You can also use single VLAN for your case and it should work fine. You can follow the example given in the link you shared.

Let me know if you have any questions.

Regards,

Kanwal

[Pathirage Don Amal Rajapakse]

Dear Kanwal,

Thanks my friend, now I have a better confident for my ACE cluster deployment. Again thanks for your kind help to a newbie like me. I wish for the success of your future carrier life.

Regards...!

-Amal-

[Kanwaljeet Singh]

Hi Amal,

Happy to help. If you have any questions please let us know.

Regards,

Kanwal

[Pathirage Don Amal Rajapakse]

Dear Kanwal,

Here I'm requesting quick help from you. I have depolyed my ACE cluster successfully. So basically what I did was, first I configured management and HA clsuter via CLI, then on wards I moved to GUI (web based) and continued my configuration. I deployed cluster in "One-Arme" mode and finally after I finished the configuration part it was working fine and was able to take back-up configuration via CLI too. But it was only remained for two days, suddenly I realized I cannot ping (VIP / MGT) or even log into devices via GUI. But I was able to telnet to active device (Admin context) but in there except of HA cluster and MGT configurations all other previous config that I have done were lost. I was wondering why it happned that way ? Does it cause since I configured via GUI ?  Or is it common problem for these devices ?  Anyway I have attached the fresh config just after I took, I finished the deployement from the device (active one) for your review.

no ft auto-sync startup-config

boot system image:c4710ace-t1k9-mz.A5_1_2.bin

peer hostname EBSLB2

hostname EBSLB1

interface gigabitEthernet 1/1

  channel-group 1

  no shutdown

interface gigabitEthernet 1/2

  channel-group 1

  no shutdown

interface gigabitEthernet 1/3

  shutdown

interface gigabitEthernet 1/4

  shutdown

interface port-channel 1

  switchport trunk native vlan 200

  switchport trunk allowed vlan 455,999

  no shutdown

ntp server 192.168.1.20

access-list ALL line 8 extended permit ip any any

probe http Get-Method

  description Check to url access /OA_HTML/OAInfo.jsp

  port 80

  request method get url /OA_HTML/OAInfo.jsp

  expect status 200 200

probe udp http-8000-iRDMI

  description IRDMI (HTTP - 8000)

  port 8000

probe http http-probe

  description HTTP Probes

  interval 10

  faildetect 2

  passdetect interval 30

  passdetect count 2 

  request method get url /index.html

  expect status 200 200

probe icmp icmp-probe

  description ICMP PROBE FOR TO CHECK ICMP SERVICE

rserver host ebsapp1

  description ebsapp1.xxxx.lk

  ip address 172.25.45.19

  conn-limit max 4000000 min 4000000

  probe icmp-probe

  probe http-probe

  inservice

rserver host ebsapp2

  description ebsapp2.xxxx.lk

  ip address 172.25.45.20

  conn-limit max 4000000 min 4000000

  probe icmp-probe

  probe http-probe

  inservice

serverfarm host ebsppsvrfarm

  description ebsapp server farm

  failaction purge

  probe http-probe

  probe icmp-probe

  inband-health check log 5 reset 500

  retcode 404 404 check log 1 reset 3

  rserver ebsapp1 80

    conn-limit max 4000000 min 4000000

    probe icmp-probe

    inservice

  rserver ebsapp2 80

    conn-limit max 4000000 min 4000000

    probe icmp-probe

    inservice

class-map type http loadbalance match-any default-compression-exclusion-mime-type

  description DM generated classmap for default LB compression exclusion mime types.

  2 match http url .*gif

  3 match http url .*css

  4 match http url .*js

  5 match http url .*class

  6 match http url .*jar

  7 match http url .*cab

  8 match http url .*txt

  9 match http url .*ps

  10 match http url .*vbs

  11 match http url .*xsl

  12 match http url .*xml

  13 match http url .*pdf

  14 match http url .*swf

  15 match http url .*jpg

  16 match http url .*jpeg

  17 match http url .*jpe

  18 match http url .*png

class-map match-all ebsapp-vip

  2 match virtual-address 172.25.45.21 tcp eq www

class-map type management match-any remote_access

  2 match protocol xml-https any

  3 match protocol icmp any

  4 match protocol telnet any

  5 match protocol ssh any

  6 match protocol http any

  7 match protocol https any

  8 match protocol snmp any

policy-map type management first-match remote_mgmt_allow_policy

  class remote_access

    permit

policy-map type loadbalance first-match ebsapp-vip-l7slb

  class default-compression-exclusion-mime-type

    serverfarm ebsppsvrfarm

  class class-default

    serverfarm ebsppsvrfarm

    compress default-method deflate

policy-map multi-match int455

  class ebsapp-vip

    loadbalance vip inservice

    loadbalance policy ebsapp-vip-l7slb

    loadbalance vip icmp-reply active

    nat dynamic 1 vlan 455

interface vlan 455

  ip address 172.25.45.36 255.255.255.0

  access-group input ALL

  nat-pool 1 172.25.45.22 172.25.45.22 netmask 255.255.255.0 pat

  service-policy input remote_mgmt_allow_policy

  service-policy input int455

  no shutdown

ft interface vlan 999

  ip address 10.1.1.1 255.255.255.0

  peer ip address 10.1.1.2 255.255.255.0

  no shutdown

ft peer 1

  heartbeat interval 300

  heartbeat count 10

  ft-interface vlan 999

ft group 1

  peer 1

  priority 110

  associate-context Admin

  inservice

ip route 0.0.0.0 0.0.0.0 172.25.45.1

username admin password 5 $1$zpY3IzVA$riLgdSwBGh5JNi3wPfoNO.  role Admin domain

default-domain

username www password 5 $1$unWErra6$0asJJtjiBp2wBbwPbLOSs.  role Admin domain default-domain

Thanks.....!

-Amal-

[Kanwaljeet Singh]

Hi Amal,

Why have you configured "no ft auto-sync startup-config", this will prevent the auto synchronization of start-up configuration. Please do ft auto-sync startup-config.  Did your ACE reload ? If you did wr mem or copy run-conig startup-config , i don't see a reason why you should lose config.

Regards,

Kanwal

[Pathirage Don Amal Rajapakse]

Dear Kanwal,

I got another problem, when I was testing cluster fail over. When I power down active ACE, my standby ACE come up and become active (I can check it via CLI) but my VIP cannot ping. Whicc means my virtual server will down. Then after I power on the first ACE then my VIP will get back on up state. What do you think about this ? (You can see my active config above)

Thanks...!

-Amal-

[Kanwaljeet Singh]

Hi Amal,

Are you able to ping it from the switch? Do you see that switch updated the ARP table after the failover happened? The modules are in different chassis or single? I think it must me something simple but i would need more info. The above FT config looks fine.

Regards,

Kanwal

[Pathirage Don Amal Rajapakse]

Dear Kanwal,

Thanks for your reply. Finally I was able to figured out that issue by help of following document by Cisco.

http://www.cisco.com/web/JP/solution/places/literature/pdf/ACE_4710_HA_Configuration.pdf

So what I did was, According to above guide lines I just added following commands into my configuration

------------------------------------------------------------------------------------------------------

resource-class LoadBalancingResources

  limit-resource all minimum 0.00 maximum unlimited

  limit-resource sticky minimum 10.00 maximum equal-to-min

context Admin

  member LoadBalancingResources

------------------------------------------------------------------------------------------------------

This fixed the issue which I have mentioned in my last post

Now I have two new concerns, hope you can help me out for those.

My first concern is, I have two cluster ACE 4710 appliance, Let's say A and B.  Currently A is active one and B is stand by mode. What I wanted to is to minimize the fail over down time in my production network, Let's say A goes down then B comes up and will be active state, but when A comes back again B should remain as active state and A will stay stand by mode. So I followed the theory which means as Cisco says set priority level equal in both ACE then disable the preemt. So finally I configured like this. (Note : I have only one ft group and Admin context only)

ft group 1

  peer 1

  no preempt 

  associate-context Admin

  inservice

But unfortunetly when B is active then A comes back, A will become active state by shifting all the services from B to A. How can I address to this issue ?

My second concern is, Let's say currently A is active and I'm gonna unplug the cable of it (Trunk up link to access switch) then I can see there is only two ping "time out" goes for VIP interface and my device B will take over the active state without any issue. Then again I tried backwards, which means my B is currently active then I pulgged the cable (Trunk up link to access switch) back to device A, just after I plugged the cable to A, A takes over the active state from B (I can see it via my CLI)  but still my ping reply from VIP interface is not positive, means it takes nearly 2 mins to become up state (get the ping reply from VIP interface). How can I address to this issue ?

Thanks....!

-Amal-

[Pathirage Don Amal Rajapakse]

Dear Kanwal,

I was able to deploy ACE cluster and now it's working fine. Now through CLI I can monitor devices resource usage status, but I'm not able to see it on my GUI dash board. Those graphs are empty. What can I do for it ? Do you have any idea ? Anyway here I have attached the GUI screen shots for your reference.

Thanks...!

-Amal-

[Kanwaljeet Singh]

Hi Amal,

I am not sure about it. Haven't used and worked on GUI related issues. If you click on "Resource usage" option on the left, do you see any change? I see it is not highlighted.

I would suggest opening a TAC case and let them have a look or wait if anyone has any ideas.

Regards,

Kanwal

[Pathirage Don Amal Rajapakse]

Dear Kanwal,

Is this issue relate to SNMP configuration on ACE ? because it indicates the device status as "down" (SNMP) on GUI. You can see it on my screen shot.

Thanks...!

-Amal-

[Kanwaljeet Singh]

Hi Amal,

Sorry i missed but yes you are right. You should have SNMP configuration in place for monitoring to work.

Please look at the link below and visit sections:

Error monitoring

Graphing Data

Prerequisite

Before using the Monitoring functions, you must:

•Enable monitoring on the virtual contexts or servers (see Setting Up Virtual Contexts Statistics Collection and Monitoring Probes

•Ensure that you allow the SNMP protocol and enter the v2c community string in the Config > System > Primary Attributes page.

•Select the virtual context you want to monitor. This step is reflected in the monitoring procedures as part of selecting your task; such as Monitor > Virtual Contexts > context > Load Balancing

http://www.cisco.com/c/en/us/td/docs/app_ntwk_services/data_center_app_services/ace_appliances/vA5_1_0/configuration/device_manager/guide/dmguigd/UGconfg.html

Regards,

Kanwal