Cisco Support Community
cancel
Showing results for 
Search instead for 
Did you mean: 

Ask the Expert: Configuration and Troubleshooting the Cisco Application Control Engine (ACE) load balancer

With Ajay Kumar and Telmo Pereira 

 

Ajay Kumar Telmo PereiraWelcome to the Cisco Support Community Ask the Expert conversation. This is an opportunity to learn and ask questions about configuration and troubleshooting the Cisco Application Control Engine (ACE) load balancer with Cisco expert Ajay Kumar and Telmo Pereira. The Cisco ACE Application Control Engine Module for Cisco Catalyst 6500 Series Switches and Cisco 7600 Series Routers is a next-generation load-balancing and application-delivery solution. A member of the Cisco family of Data Center 3.0 solutions, the module: Helps ensure business continuity by increasing application availability Improves business productivity by accelerating application and server performance Reduces data center power, space, and cooling needs through a virtualized architecture Helps lower operational costs associated with application provisioning and scaling

 

Ajay Kumar  is a customer support engineer in the Cisco Technical Assistance Center in Brussels, covering content delivery network technologies including Cisco Application Control Engine, Cisco Wide Area Application Services, Cisco Content Switching Module, Cisco Content Services Switches, and others. He has been with Cisco for more than four years, working with major customers to help resolve their issues related to content products. He holds DCASI and VCP certifications. 

 

Telmo Pereira is a customer support engineer in the Cisco Technical Assistance Center in Brussels, where he covers all Cisco content delivery network technologies including Cisco Application Control Engine (ACE), Cisco Wide Area Application Services (WAAS), and Digital Media Suite. He has worked with multiple customers around the globe, helping them solve interesting and often highly complex issues. Pereira has worked in the networking field for more than 7 years. He holds a computer science degree as well as multiple certifications including CCNP, DCASI, DCUCI, and VCP

 

Remember to use the rating system to let Ajay know if you have received an adequate response.

 

Ajay and Telmo might not be able to answer each question due to the volume expected during this event. Remember that you can continue the conversation on the Data Center sub-community discussion forum Application Networking shortly after the event.

 

This event lasts through July 26, 2013. Visit this forum often to view responses to your questions and the questions of other community members.

 

 
25 REPLIES
New Member

Ask the Expert: Configuration and Troubleshooting the Cisco Appl

Hello Experts,

I am planning to upgrade my ACE and would like to know the best practices for the same?

Will there be a downtime or it can be a hitless upgrade. I think it should be simple but need your opinion. I think I can start with upgrade of standby.  If for some reason ACE doesn’t boot up what would be the recovery steps.

Appreciate your quick response.

-John

Cisco Employee

Ask the Expert: Configuration and Troubleshooting the Cisco Appl

Hi John,

Will there be a downtime or it can be a hitless upgrade.

You dont need downtime to upgrade ACE. It can be a hitless upgrade.

You can follow the procedure as described in link below:

http://www.cisco.com/en/US/docs/interfaces_modules/services_modules/ace/vA5_1_x/release/note/ACE_mod_rn_A51x.html#wp791479

From the above link :

Note  : Ensure that the preempt command is disabled before the upgrade procedure begins.

It is also true that you first upgrade standby and then Primary. The above mentioned link is the best way to upgrade ACE.

------------------------------------------------------------

If for some reason ACE does not boot up. Below is the recovery procedure:

Usually it get stuck in rommon mode:

You can refer the following link : 

http://docwiki.cisco.com/wiki/Cisco_Application_Control_Engine_(ACE)_Troubleshooting_Guide_--_Troubleshooting_ACE_Boot_Issues

refer topic : " Booting the ACE from the ROMMON Prompt"

In the above you can mention the old image and ACE should boot properly with the old ACE image.

Let me know if that answer your question.

regards,

Ajay Kumar

New Member

Ask the Expert: Configuration and Troubleshooting the Cisco Appl

Thanks Ajay.

Cisco Employee

Ask the Expert: Configuration and Troubleshooting the Cisco Appl

Hello John!!!

Thanks for the first post on this session! Just to add some additional information to what Ajay has shared.

We do have a hitless upgrade, but refers to L4 connections which can be replicated (if you do have connection replication enabled on your system).

However be aware that there will be a hit on L7 connections (SSL offload, TCP Reuse, Inspect, etc). Meaning those connections will have to be reestablished on the secondary ACE.

In that sense, if you do have any contexts with layer 7 configuration, or even if do have only L4, the general recommendation for that matter is to schedule a maintenance window for the upgrade operation.


Also as per best practice we recommend:

1. To disable preemption (as Ajay mentioned) and upgrade the standby box for the Admin context

2. Then you reboot the standby box on the new version of code and you do a failover of the contexts to that box.

You will see how the system behaves. If there are any issues, you can simply fallback to the other ACE that will still be on the old version of code.

3. Assuming everything goes well on 2, you will go ahead and upgrade the other ACE and once if comes back, you can failover the traffic to it again and reenabe preempt.

This is documented, on a step by step basis on the link provided by Ajay.

HTH,

Telmo


New Member

Ask the Expert: Configuration and Troubleshooting the Cisco Appl

Thanks Telmo.

New Member

Ask the Expert: Configuration and Troubleshooting the Cisco Appl

Dear AJay and Telmo,

Thanks for your time to look at this. We have a pair of ACE4710 running in active/standby. We've setup a web services to load balance https://www.mydomain.com/project/ to 6 backend servers. The servers could accept uri path of /project/ and /project (with or without a /).

What are the steps to configure on ACE to be able to accept traffic for both / and without /?

my second question is, do I need to configure both access-group and service-policy on all the interfaces to pass the traffic? What are the differences betweeen an access-group and a service-policy?

Thank you in advance,

James Ren

Cisco Employee

Ask the Expert: Configuration and Troubleshooting the Cisco Appl

Jing,

Thanks for posting!

In order to meet your requirements you would have a similar configuration to the one below:

If the requested URL is /project, then send to the PROJECT-FARM, otherwise send to web farm.

-- CONF snippet --

access-list ANYONE line 10 extended permit tcp any any

rserver host WWW_SERVER_01

  ip address 10.10.10.10

  inservice

rserver host WWW_SERVER_02

  ip address 10.10.10.11

  inservice

rserver host WWW_SERVER_03

  ip address 10.10.10.12

  inservice

rserver host LOGIN_SERVER_04

  ip address 10.10.10.15

  inservice

serverfarm host WWW-FARM

  probe TCP

  rserver WWW_SERVER_01

    inservice

  rserver WWW_SERVER_02

    inservice

  rserver WWW_SERVER_03

    inservice

serverfarm host PROJECT-FARM

  probe TCP

  rserver LOGIN_SERVER_04

    inservice

class-map match-all WWW-VIP

  2 match virtual-address 20.20.20.10 tcp eq www

class-map type http loadbalance match-any CLASS-PROJECT

  2 match http url /project.*

policy-map type loadbalance first-match SLB_LOGIC

  class CLASS-PROJECT

    serverfarm PROJECT-FARM

  class class-default

    serverfarm WWW-FARM

policy-map multi-match CLIENT_VIPS

  class WWW-VIP

    loadbalance vip inservice

    loadbalance policy SLB_LOGIC

    loadbalance vip icmp-reply

interface vlan 10

  description Servers vlan

  access-group input ANYONE

  ip address 10.10.10.5 255.255.255.0

  no shutdown

interface vlan 20

  description Client vlan

  ip address 20.20.20.9 255.255.255.0

  access-group input ANYONE

  service-policy input CLIENT_VIPS

  no shutdown

ip route 0.0.0.0 0.0.0.0 20.20.20.1

For the second query, you just need to be aware that by default ACE will not pass any traffic (will deny everything), unless we permit it.

The access-group will be used to tie the access-list to an interface and this is normally needed on all interfaces if you are going to pass any sort of traffic on them.

The service-policy may or not be needed on all interfaces. In the example I gave you, I only apply service-policy on the client vlan as I will only have hits for the VIP on that side.

If on the other hand, servers could also initiate connections to the VIP, you would have to apply the service policy on that other interface also.

Let me know if this makes things clearer, if not I can provide more details.

HTH,

Telmo

New Member

Ask the Expert: Configuration and Troubleshooting the Cisco Appl

Thanks for your explanation and detailed config sample. I helped a lot for me to understand the concept but it might not be applicable to our environment. We have a multi-tenant load balancing setup and multiple class-maps are applied. Futhermore, class-default has already been used for some other servers, which is a different sticky server farm.

Is it possible to set a uri redirect, similar to an iRule in F5 to force client browser to go to /project/ if they are coming without a / (force /project to /project/) ?

The current setup is

2 match http url /project.*

which works fine for /project/ but it does not respond to /project

Your explanation on the access-group and policy-map is very clear. Thank you!

Best Wishes,

James Ren

Cisco Employee

Ask the Expert: Configuration and Troubleshooting the Cisco Appl

James,

The pleasure is all mine!

It is possible to configure URL redirection on ACE, but it seems this would purely polute your configuration unnecessarily.

  2 match http url /project.*

Should match both /project and /project/.

Actually it should also match URLs like /project/anythingelse.something

.* is an expression that is supposed to match zero or more characters.

It would come as a suprise if it is not matching the request for /project. Is the request HTTP/1.1? What ACE version do you have?

Anyway, to give you an example of what a redirection would look like on ACE here it goes:

class-map type http loadbalance match-any redirect-l7

  2 match http url /project

policy-map type loadbalance first-match redirect-policy

  class redirect-l7

    serverfarm redirect-sf

serverfarm redirect redirect-sf

  rserver redirect-sf

    inservice

rserver redirect redirect-sf

  webhost-redirection http://%h/project/ 301

  inservice

%h represents the hostname.

This has been documented here:

http://www.cisco.com/en/US/docs/interfaces_modules/services_modules/ace/v3.00_A2/configuration/slb/guide/rsfarms.html#wp1013201

Still, as mentioned, I don't see a need to apply redirection on this case. If "match http url /project.*" is not working for some reason just let me know and I will advise on next steps.


Regards,

Telmo

New Member

Re: Ask the Expert: Configuration and Troubleshooting the Cisco

Dear Telmo,

Sorry for the late reply. I was caught up with something else.

Your explanation on the match-all regex is what I thought would work. But in fact it didn't work as intended.

I have tested on the a random backend server and they accept with or without /.

I tested on the frontend, / works but without a / it doesn't.

Please refer to the attached screenshots.

Below is the relevant configuration:-

class-map type http loadbalance match-all CLASS_MCTP

  2 match http url /.*

policy-map type loadbalance http first-match APP

  class CLASS_MCTP

    sticky-serverfarm STICKY_MCTP

    action SSL_REWRITE_MCTP

policy-map multi-match POLICY

  class xxxxxxxd11_443_CLASS

loadbalance vip inservice

    loadbalance policy APP

    loadbalance vip icmp-reply active

    appl-parameter http advanced-options CASE_PARAM

    ssl-proxy server xml_xxxx

action-list type modify http SSL_REWRITE_MCTP

  ssl url rewrite location ".*xml\.mydomain\.com.*" clearport 9380

Cisco Employee

Re: Ask the Expert: Configuration and Troubleshooting the Cisco

Dear James,

Welcome back to the thread.

The issue is actually your configuration, you are matching at least one ´/´ on the URL, if it is not there it will not work, meaning ACE will not match the policy for loadbalancing, and as you don't have a class class-default as fallback it will simply fail.

Can you please try to change the class-map CLASS_MCTP to

class-map type http loadbalance match-all CLASS_MCTP

  2 match http url /mctp.* 

That should do the trick.

If you want something even more generic you can use:

class-map type http loadbalance match-all CLASS_MCTP

  2 match http url .* 

HTH,

Telmo

Cisco Employee

Ask the Expert: Configuration and Troubleshooting the Cisco Appl

Hi Jing,

Just to add a bit on Telmo's reply.

What are the differences betweeen an access-group and a service-policy?

We apply access group to any interface to define what traffic should be allowed or dropped when hitting a interface. Similar to any firewall access list/access group.

Service policy is like a set of instruction to match the interesting traffic based on defined associated class match and to define how to load balance the traffic. So I would say that load balancing decisions are based on service policy.

regards,

Ajay Kumar

New Member

Ask the Expert: Configuration and Troubleshooting the Cisco Appl

Hello, Expert.

In the case that two servers are on the each end sides of firewall (one on DMZ and another on INSIDE), is there a way to load-balance the traffic from one to another server between firewall A and B, using the ACE 4710? Or, there is a way to load-balance the traffic using source port number?

Thank you for reading it.

Cisco Employee

Ask the Expert: Configuration and Troubleshooting the Cisco Appl

Hello Jeongdae,

Thanks for the question.

As per my understanding you want to load balance in the following way :

Firewall DMZ ( Server in DMZ zone acting as client ) >> Cisco ACE >> Firewall Inside ( Server in INSIDE zone ) 

This is usual load balancing scenerio in routing mode. This can be achieved by simple routing mode config:

Please refer following link for routing mode config.

http://docwiki.cisco.com/wiki/Basic_Load_Balancing_Using_Routed_Mode_on_the_Cisco_Application_Control_Engine_Configuration_Example

It is possible to load balance using source IP address.

Sample config :

class-map match-any TEST-VIP
  3 match virtual-address x.x.x.x eq any

serverfarm SF1

  rserver A
    inservice

  rserver B

    inservice

class-map type http loadbalance match-any SRC-IP-MATCH
  2 match source-address a.a.a.a 255.255.255.255

policy-map type loadbalance first-match Policy1
  class SRC-IP-MATCH
    serverfarm SF1

policy-map multi-match Mpolicy1

  class TEST-VIP

    loadbalance vip inservice
    loadbalance policy Policy1
    loadbalance vip icmp-reply
    loadbalance vip advertise

interface vlan yyy

service-pilicy .....

Let me know if that helps.

regards,

Ajay Kumar

New Member

Ask the Expert: Configuration and Troubleshooting the Cisco Appl

Hello Ajay.

Thanks for reply.

My network diagram is the following :

Server1---->ACE1 ----> ASA1----->ACE2----->Server2

                            | ------>ASA2----------|

        "DMZ"                        |                        "INSIDE"

Two servers are communicating with each other via specific tcp port numbers.

My problem is how to load-balance the traffic from server1 to server2 and vice versa through both ASA1 and ASA2.

Currently, the ASA1 only has the connections from server1 to server2.

Thank you for reading.

Cisco Employee

Ask the Expert: Configuration and Troubleshooting the Cisco Appl

Hello Jeongdae,

I understand it now. You can refer the following link to do firewall load balancing.

http://www.cisco.com/en/US/docs/app_ntwk_services/data_center_app_services/ace_appliances/vA3_1_0/configuration/slb/guide/fwldbal.html

mac-sticky enable command is the one which does the trick here. It keeps the session with the same firewall.

let me know if you some specific question related to this setup.

regards,

Ajay Kumar

New Member

Ask the Expert: Configuration and Troubleshooting the Cisco Appl

Dear Experts,

I'd like to ask you about configuration amendments for using sorry serverfarm (sorry page when primary sfarm fails) with regard to this specific example:

rserver host REAL_PRIMARY_1

  ip address 10.0.0.1

  inservice

rserver host REAL_PRIMARY_2

  ip address 10.0.0.2

  inservice

rserver host REAL_BACKUP_1

  ip address 10.0.0.11

  inservice

rserver host REAL_BACKUP_2

  ip address 10.0.0.12

  inservice

serverfarm host SFARM_PRIMARY

  failaction reassign

  predictor leastconns slowstart 30

  probe PROBE_HTTP

  rserver REAL_PRIMARY_1 80

    conn-limit max 10000 min 9900

    inservice

  rserver REAL_PRIMARY_2 80

    conn-limit max 10000 min 9900

    inservice

serverfarm host SFARM_BACKUP

  failaction reassign

  predictor leastconns slowstart 30

  probe PROBE_HTTP

  rserver REAL_BACKUP_1 80

    inservice

  rserver REAL_BACKUP_2 80

    inservice

sticky ip-netmask 255.255.255.255 address source STICKY_PRIMARY

  timeout 120

  serverfarm SFARM_PRIMARY backup SFARM_BACKUP

sticky http-cookie BACKUP_ID STICKY_BACKUP

  cookie insert

  timeout 60

  serverfarm SFARM_BACKUP

action-list type modify http ACT_LIST_RW

  ssl url rewrite location ".*"

policy-map type loadbalance first-match PM_L7_PRIMARY

  class class-default

    sticky-serverfarm STICKY_PRIMARY

    action ACT_LIST_RW

policy-map type loadbalance first-match PM_L7_BACKUP

  class class-default

    sticky-serverfarm STICKY_BACKUP

class-map match-all CM_L3L4_PRIMARY

  10 match virtual-address 10.0.0.100 tcp eq https

class-map match-all CM_L3L4_BACKUP

  10 match virtual-address 10.0.0.200 tcp eq www

parameter-map type connection PARA_MAP_CONN_TIMEOUT_120

  set timeout inactivity 120

parameter-map type http PARA_MAP_PERSIST_REBAL

  persistence-rebalance

  header modify per-request

policy-map multi-match PM_L3L4_PRIMARY

  class CM_L3L4_PRIMARY

    loadbalance vip inservice

    loadbalance policy PM_L7_PRIMARY

    loadbalance vip icmp-reply active

    appl-parameter http advanced-options PARA_MAP_PERSIST_REBAL

    ssl-proxy server SSL_PROXY_PRIMARY

    connection advanced-options PARA_MAP_CONN_TIMEOUT_120

policy-map multi-match PM_L3L4_BACKUP

  class CM_L3L4_BACKUP

    loadbalance vip inservice

    loadbalance policy PM_L7_BACKUP

    loadbalance vip icmp-reply active

    appl-parameter http advanced-options PARA_MAP_PERSIST_REBAL

interface vlan 100

  description Client-side VLAN

  bridge-group 100

  mac-sticky enable

  access-group input ACL_ALLOW_BPDU

  access-group input ACL_ALL_IP

  access-group output ACL_ALL_IP

  service-policy input PM_L3L4_PRIMARY

  service-policy input PM_L3L4_BACKUP

  no shutdown

interface vlan 200

  description Server-side VLAN

  bridge-group 100

  mac-sticky enable

  access-group input ACL_ALLOW_BPDU

  access-group input ACL_ALL_IP

  access-group output ACL_ALL_IP

  no shutdown

interface bvi 100

  ip address 10.0.0.252 255.255.255.0

  alias 10.0.0.254 255.255.255.0

  peer ip address 10.0.0.253 255.255.255.0

  no shutdown

My questions for the above example:

1) After making the first test, it seems that users who want to connect to a  failed primary page, then they see the sorry page. However, after  the primary sfarm is up, the users still see sorry page. What can cause that behaviour? Should I remove http cookie for the backup sfarm?

2) How will failaction reassign work, when the primary sfarm will go down reaching the MAXCONN state?

3) How will the ACE behave, when backup sfarm is configured under sticky of primary sfarm? Connections only for backup sfarm will be added into the sticky database?

4) Is there something else to configure for sorry sfarm if the requirement is to configure backup sfarm as host not as a redirect sfarm (from cisco.com - there is an example for configuring sorry sfarm as redirect)?

Kind of regards,

Krzysztof

Cisco Employee

Ask the Expert: Configuration and Troubleshooting the Cisco Appl

Krzysztof,

Here it goes the answers to your queries:

1) This is expected behavior. Once the connections are on the backup serverfarm, they will stay there until they complete. But as you are suspecting stickiness also plays a role here. So assuming you have sticky for the backup serverfarm this is the behavior you should see:

–All new sticky connections that match existing sticky table entries for the real servers in the backup server farm are stuck to the same real servers in the backup server farm.

–All new non-sticky connections and those sticky connections that do not have an entry in the sticky table are load balanced to the real servers in the primary server farm.

Hopefully this is what you are seeing

2) When you reach the MAXCONN threshold, you can expect that new connections will be sent to the backup farm. Depending on the platform you are using, this threshold value may be divided by the amount of IXPs (Network processors).

3) That is normally what we see customers doing, so in spite of having:

sticky ip-netmask 255.255.255.255 address source STICKY_PRIMARY

timeout 120

serverfarm SFARM_PRIMARY backup SFARM_BACKUP

sticky http-cookie BACKUP_ID STICKY_BACKUP

  cookie insert

timeout 60

serverfarm SFARM_BACKUP

You would simply do (note the sticky keyword after the backup farm):

sticky ip-netmask 255.255.255.255 address source STICKY_PRIMARY

  timeout 120

  serverfarm SFARM_PRIMARY backup SFARM_BACKUP sticky

And the behavior is exactly the same. If all the servers in the primary server farm go down, the ACE sends all new requests to the backup server farm. When the primary server farm comes back up (at least one server becomes active):

•If the sticky option is enabled, then:

–All new sticky connections that match existing sticky table entries for the real servers in the backup server farm are stuck to the same real servers in the backup server farm.

–All new non-sticky connections and those sticky connections that do not have an entry in the sticky table are load balanced to the real servers in the primary server farm.

•If the sticky option is not enabled, then the ACE load balances all new connections to the real servers in the primary server farm.

•Existing non-sticky connections to the servers in the backup server farm are allowed to complete in the backup server farm.

This has been documented here:

http://www.cisco.com/en/US/docs/interfaces_modules/services_modules/ace/vA2_3_0/configuration/slb/guide/sticky.html#wp1137791

4) No, at first sight your configuration looks fine. However if I understood you correctly, you would need to remove sticky for the backup farm to meet your requirements. Or at least to achieve a behavior close to what you are expecting.

HTH,

Telmo

New Member

Ask the Expert: Configuration and Troubleshooting the Cisco Appl

Hello Telmo,

Thank you very much for your answers

Here are my conclusions:

1) & 3) As suspected, the stickiness caused the problem with users when the sorry page was triggered by them. I am about to remove the sticky http-cookie from the backup sfarm but what about this sentence from the link that you provided (the last paragraph under that section):

If you want to configure sorry servers and you  want existing connections to revert to the primary server farm after it  comes back up, do not use stickiness.

Does it mean to not use stickiness for primary and backup sfarm or only for backup sfarm?

2) I'm using ACE module on Cisco 6500 switch, IOS ver: A2(3.5) and there are 2 NP's. When using sorry sfarm, should I also remove failaction reassign? This action is rather used for passing traffic to stateful firewalls (as backups). Please correct me if I'm wrong

4) Regardless of understanding the above sentence from ACE config guide (I guess, they had in mind backup sfarm only), removing sticky from backup sfarm will solve the problem

Much appreciated for your help,

Krzysztof

Cisco Employee

Ask the Expert: Configuration and Troubleshooting the Cisco Appl

1) & 3) That sentence is only in reference to the backup farm.

2) You are correct. Failaction reassign will simply begin sending packets to the remaining/backup rserver.  There must be some logic to sync the state on the OS/App across all of the rservers for this to work properly or the new server will simply drop the connection, since it never saw the handshake.  Typically you see this feature used with firewall load

balancing because firewalls are replicating their connection state tables.

4) Correct.

Pleasure is all mine, hope this helps.

Regards,

Telmo

New Member

Ask the Expert: Configuration and Troubleshooting the Cisco Appl

Thank you Telmo.

It really helped me

New Member

Ask the Expert: Configuration and Troubleshooting the Cisco Appl

Hi Mr. Ajay am desperately in need of your help in configuring a cisco 3620 series router for VPN

remote access with ios

flash:c3620-jk9s-mz.122-29.bin please help my boss is on me for this please please help

Cisco Employee

Ask the Expert: Configuration and Troubleshooting the Cisco Appl

Hi Gilbert,

I am sorry but this is not my area of expertise. I deal with Cisco ACE.  You can post the same question in security support forum, but try to provide more details and requirements so that they can help you out.

regards,

Ajay Kumar

New Member

Re: Ask the Expert: Configuration and Troubleshooting the Cisco

Hello Experts,

I'd like to ask you, how to interpret the values from resource usage in high details:

ACE/Context# show resource usage
                                                     Allocation
        Resource         Current       Peak        Min        Max       Denied
-------------------------------------------------------------------------------
-- outputs omitted for brevity --
  proxy-connections             0      16358      16358      16358      17872
  ssl-connections rate          0        626        626        626      23204

Some tests were done with many SSL connections per second (more than the limit).

1) What would the ACE normally do with denied SSL connections? Could it be possible if the ACE will send to clients TCP Resets? (during the tests TCP Resets were captured)

2) What is proxy-connections shown in the resources? And the same question as above - what the ACE can do with denied proxy conns? What TCP flags will be send to clients?

I guess to overcome the high number of denied connections - there will be a need to increase values under the resource class.

3) If sticky is configured for a sfarm with conn-limit set, and if a client will initiate traffic to the sfarm which will be UP (conn-limit not yet reached) - sticky will add source client ip and associated rserver (as configured sticky method).

What will happen if the client will lose connection but the client still will have existing sticky entry (2 hours set for existence of sticky) and now, the client will try to generate traffic to sfarm which is now DOWN due to MAXCONN state on all rservers?

Sticky has no relations with current conns on rservers?

Best regards,

Krzysztof

Cisco Employee

Re: Ask the Expert: Configuration and Troubleshooting the Cisco

Hello Krzysztof,

Another set of good/interesting questions posted. Thanks! 

I will try to clarify your doubts.

In the output below both resources (proxy-connections and ssl-connections rate) are configured with a min percentage of resources (column Min), while 'Max' is set to equal to the min.

ACE/Context# show resource usage

                                                     Allocation

        Resource         Current       Peak        Min        Max       Denied

-------------------------------------------------------------------------------

-- outputs omitted for brevity --

  proxy-connections             0      16358      16358      16358      17872

  ssl-connections rate          0        626        626        626      23204

Most columns are self explanatory, 'Current' is current usage, 'Peak' is the maximum value reached, and the most important counter to monitor 'Denied' represents the amount of packets denied/dropped due to exceeding the configured limits.

On the resources themselves, Proxy-connections is simply the amount of proxied connections, in other words all connections handled at layer 7 (SSL connections are proxied, as are any connections with layer 7 load balance policies, or inspection).

So in this particular case for the proxy-connections we see that Peak is equal to the Max allocated, and as we have denies we can conclude that you have surpassed the limits for this resource. We see there were 17872 connections dropped due to that.

ssl-connections rate should be read in the same manner, however all values for this resource are in bytes/s, except for Denied counter, that is simply the amount of packets that were dropped due to exceeding this resource. 

For your particular tests you have allocated a min percentage and set max equal to min, this way you make sure that this context will not use any other additional resources.

If you had set the max to unlimited during resource allocation, ACE would be allowed to use additional resources on top of those guaranteed, if those resources were available.

This might sound a great idea, but resource planning on ACE should be done carefully to avoid any sort of oversubscription, specially if you have business critical contexts.

We have a good reference for ACE resource planning that contains also description of all resources (this will help to understand the output better):

http://www.cisco.com/en/US/docs/interfaces_modules/services_modules/ace/v3.00_A2/configuration/virtualization/guide/config.html#wp1008224

1) When a resource is utilized to its maximum limit, the ACE denies additional requests made by any context for that resource. In other words, the action is to Drop. ACE  should in theory silently drop (No RST is sent back to the client). So unless we changed something on the code, this is what you should see.

To give more context, seeing resets with SSL connections is not necessarily synonym of drops. As it is usual to see them during normal transactions.

For instance Microsoft servers are usually ungracefully terminating SSL connections with RESET. Also when there is renegotiation during an SSL transaction you may see RESETS, but this will pass unnoticed for end users. 

2)  ACE will simply drop/ignore new connections when we reach the maximum amount of proxied connections for that context. Exisiting connections will continue there.

As ACE doesn't respond back, client would simply retransmit, and if he is lucky maybe in the next attempt he will be able to establish the connection.

To overcome the denies, you will definitely have to increase the resource allocation. This of course, assuming you are not reaching any physical limit of the box.

As mentioned setting max as unlimited might work for you, assuming there are a lot of unused resources on the box.

3)  If a new connection comes in with a sticky value, that matches the sticky entry of a real server, which is already in MAXCONNS state, then both the ACE module/appliance should reject the connection and that sticky entry would be removed.

The client would at that point reestablish a new connection and ACE would associate a new sticky entry with the flow for a new RSERVER after the loadbalancing decision.

I hope this makes things clearer! Uff...

Regards,

Telmo

4432
Views
70
Helpful
25
Replies
CreatePlease to create content