Cisco CSS 11503 Arrowpoint/Load Balance question

Answered Question
Feb 11th, 2010

I am troubleshooting an issue with my 11503.  I am running version 07.40.0.04. I have it configured as follows:

  content upcadtoa-rule
    add service cadtoa-wls1-e0
    add service cadtoa-wls1-e1
    add service cadtoa-wls2-e0
    add service cadtoa-wls2-e1
    add service cadtoa-wls3-e0
    add service cadtoa-wls3-e1
    add service cadtoa-wls4-e0
    add service cadtoa-wls4-e1
    add service cadtoa-wls5-e0
    add service cadtoa-wls5-e1
    add service cadtoa-wls6-e0
    add service cadtoa-wls6-e1
    arrowpoint-cookie expiration 00:00:15:00
    protocol tcp
    port 8001
    advanced-balance arrowpoint-cookie
    redundant-index 2
    vip address 172.30.194.195 range 2
    arrowpoint-cookie name TOA
    active

However, the load-balancing across the servers does not seem to be doing much balancing.  One of those servers is getting hit with 5 times as much traffic as another and another server is lucky to get a connection at all.  With the cookie expiration set, one would think that this would all balance out over time.

I just came across this information from Cisco and I am wondering if it is relevant:

If you configure a balance or advanced-balance method on a content rule that requires the TCP protocol for Layer 5 (L5) spoofing, you should configure a default URL string, such as url "/*". The addition of the URL string forces the content rule to become an L5 rule and ensures L5 load balancing or stickiness. If you do not configure a default URL string, unexpected results can occur.

In the following configuration example, if you configure a Layer 3 (L3) content rule with an L5 balance method, the CSS performs L5 load balancing, but will reject UDP packets.

content testing

vip address 192.168.128.131

add service s1

balance url

active

The balance url method is an L5 load-balancing method in which the CSS must spoof the connection and examine the HTTP GET content request to perform load balancing. The CSS rejects the UDP packet sent to this rule because a UDP connection cannot be L5. Though the CSS allows this rule configuration, its expected behavior would be more clear if you promote the rule to L5 by configuring the url "/*" command.

In the next example, if you configure an L3 content rule with an L5 advanced-balance method, L5 stickiness will not work as expected.

content testing

vip address 192.168.128.131

add service s1

advanced-balance arrowpoint-cookie

active

The advanced-balance arrowpoint-cookie method causes the CSS to spoof the connection, however, the CSS still marks it as an L3 rule. Thus, the CSS does not insert the generated cookie and the rule defaults to L3 stickiness (sticky-srcip). You must configure a URL like url "/*" to promote this rule to L5, ensuring that L5 stickiness works as expected.

Thanks in advance for any help you can give.  The thing is not down, it is just balancing strangely causing application performance issues.


James

I have this problem too.
0 votes
Correct Answer by Sean Merrow about 6 years 9 months ago

Hey James,

You will need to suspend the content rule in order to add the url statement.  This will cause a quick downtime until the content rule is activated again.  I have shown below the commands to add the statement.  Perhaps you can create your commands in a Notepad file, then paste them all in so they execute quickly to minimize your downtime:

  content MY-SITE
    vip address 10.201.130.140
    port 80
    protocol tcp
    add service MY-SERVER
    active


CSS11503# config t

CSS11503(config)# owner TEST

CSS11503(config-owner[TEST])# content MY-SITE

CSS11503(config-owner-content[TEST-MY-SITE])# url "/*"
%% Attribute may not be modified on active rule

CSS11503(config-owner-content[TEST-MY-SITE])# suspend

CSS11503(config-owner-content[TEST-MY-SITE])# url "/*"

CSS11503(config-owner-content[TEST-MY-SITE])# active

CSS11503(config-owner-content[TEST-MY-SITE])# exit

CSS11503(config-owner[TEST])# exit

CSS11503(config)# exit

CSS11503# show run
    :
    :
  content MY-SITE
    vip address 10.201.130.140
    add service MY-SERVER
    port 80
    protocol tcp
   url "/*"       <--------
    active

Hope this helps,

Sean

  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 5 (1 ratings)
Loading.
Sean Merrow Thu, 02/11/2010 - 09:32

Hi James,

I would recommend that you add the url "/*" to your content rule as the documentation describes.  Also, I would highly recommend that you upgrade to the latest 8.20 software on CCO.  The image you are running is the very first release of the 7.40 release and is about 6 years old and was EoL'd in Oct. 2005.  Engineering stopped supporting it years ago and there have been hundreds of bugs since then.  That said, Cisco TAC will support you on that software until April 28, 2011.

If the url statement or the upgrade (if it is possible for you to upgrade), does not resolve the issue, then the next step would be to start gathering network captures to see what is going on.

Hope this helps,

Sean

jfraasch Thu, 02/11/2010 - 09:35

Thanks Sean.


Unfortunately, there is no SmartNet for this device.  If the upgrade in software flubs then I would be in a bit of trouble.

I think the URL command might do the trick.  It is doing load balancing now but it is nowhere near an even distribution. It shouldnt be like a round robin because of the way I am balancing with cookies, but it should be much more even than it currently is.

Do you know if adding the URL command will take the service down? Can I do it during production?


Also, what is the exact command? I have seen it in the cisco doc above but in other areas like SSL I see the "*" wildcard used.

James

Correct Answer
Sean Merrow Thu, 02/11/2010 - 09:54

Hey James,

You will need to suspend the content rule in order to add the url statement.  This will cause a quick downtime until the content rule is activated again.  I have shown below the commands to add the statement.  Perhaps you can create your commands in a Notepad file, then paste them all in so they execute quickly to minimize your downtime:

  content MY-SITE
    vip address 10.201.130.140
    port 80
    protocol tcp
    add service MY-SERVER
    active


CSS11503# config t

CSS11503(config)# owner TEST

CSS11503(config-owner[TEST])# content MY-SITE

CSS11503(config-owner-content[TEST-MY-SITE])# url "/*"
%% Attribute may not be modified on active rule

CSS11503(config-owner-content[TEST-MY-SITE])# suspend

CSS11503(config-owner-content[TEST-MY-SITE])# url "/*"

CSS11503(config-owner-content[TEST-MY-SITE])# active

CSS11503(config-owner-content[TEST-MY-SITE])# exit

CSS11503(config-owner[TEST])# exit

CSS11503(config)# exit

CSS11503# show run
    :
    :
  content MY-SITE
    vip address 10.201.130.140
    add service MY-SERVER
    port 80
    protocol tcp
   url "/*"       <--------
    active

Hope this helps,

Sean

jfraasch Thu, 02/11/2010 - 09:55

Perfect, thank you.


Since I have multiple services it will not create an outage.


James\
|
EDIT


I just realized you have to make it to the rule and not the service...so yes, you are right.  There will be a moment or two of downtime.


James

Actions

This Discussion