ASK THE EXPERT - APPLICATION CONTROL ENGINE

Unanswered Question
Oct 23rd, 2009

Welcome to the Cisco Networking Professionals Ask the Expert conversation. This is an opportunity to learn how to configure and troubleshoot the Application Control Engine with Cisco expert Gilles Dufour. Gilles is a software engineer for the Level 4 to Level 7 switches in the Internet Systems Business Unit since January 2005. Prior to this position, he was a Technical Assistance Center (TAC) customer support engineer. During his first two years at Cisco Systems, Inc. he worked as a routing protocol expert. Later on, he worked as a network consulting engineer in Denver Colorado and as an engineer in the content networking team. He is a CCIE # 3878 in routing, switching and security.

Remember to use the rating system to let Gilles know if you have received an adequate response.

Gilles might not be able to answer each question due to the volume expected during this event. Our moderators will post many of the unanswered questions in other discussion forums shortly after the event. This event lasts through November 6, 2009. Visit this forum often to view responses to your questions and the questions of other community members.

I have this problem too.
0 votes
  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 4.3 (16 ratings)
Loading.
cscherb Sun, 10/25/2009 - 12:34

HI Gilles,

ACE appliance is using user "DM" in syslog messeges when changes are made via web-interface. Is there any way to change this behaivior so the the user authenticated to the web-interface is mentioned in syslog messages ?

Best Regards

Carsten

Gilles Dufour Mon, 10/26/2009 - 06:32

Carsten,

currently this is not possible, but there is a ddts open to track this feature request : CSCsu85286.

I believe the target is the next release A3(2.5).

Regards,

Gilles.

Hi Gilles

With regards to the ACE module, we have a requirement to add a static ARP entry on the ACE. I am getting the following error message.

Error: Invalid MAC address

I'm trying to add a unicast IP address and a multicast mac-address so I'm wondering if unlike a 6509 whether or not the ACE just doesn't support it?

Are you able to say if this is supported on the ACE.

Thanks

MJ

Gilles Dufour Mon, 10/26/2009 - 06:33

we do not support multicast mac-address on the ACE module.

This is a limitation of some hardware component used the build the board.

So there is no plan to support this in a future release.

Regards,

Gilles.

mruuth Mon, 10/26/2009 - 07:26

Gilles,

Is it true, that ACE 4710 doesn't forward MST BPDU's.

Regards

Mats

Gilles Dufour Tue, 10/27/2009 - 03:42

This is true for the module.

However, I'm not 100% sure for the Appliance.

So if you can send me a sniffer trace with the type of BPDU you want to bridge, I can replay the trace in my lab and see if they go through.

Regards,

Gilles.

mruuth Fri, 11/06/2009 - 05:46

Gilles,

I have used the NAM to sniff but I didn't manage to just filter out STP, so ther are some other traffic in the attachment.

Regards

Mats

Attachment: 

Giles,

I attached a config that I am soon going to implement in our ACE module. The class-map is configured to listen to ports 80 and 90. There are two probes in the server-farm, one probing port 80 and one probing port 90.

My question is, if one of the probes fails on port 80, will the probe take the server out of rotation even though the server is still listening on port 90. I have attached the config.

Regards,

Johnny...

Attachment: 
Gilles Dufour Tue, 10/27/2009 - 03:46

Yes, with your current config if one probe fails the server is put outofservice.

You can modify this behavior with the command "fail-on-all Fail reals when all probes fail" under the serverfarm.

You could also split the class-map in 2. One for each listening port and create a serverfarm for each port/probe as well so that you can dissociate both ports from each other.

Gilles.

Gilles Dufour Tue, 10/27/2009 - 06:42

class-map map1

match virtual x.x.x.x tcp eq 80

class-map map2

match virtual x.x.x.x tcp eq 90

serverfarm farm1

probe port80

rserver server1 80

ins

serverfarm farm2

probe port90

rserver server1 90

ins

policy type load first Pol1

class class-default

serverfarm farm1

policy type load first Pol2

class class-default

serverfarm farm2

policy multimatch SLB

class map1

load policy Pol1

class map2

load policy Pol2

G.

UHansen1976 Tue, 10/27/2009 - 15:07

Hi Gilles,

First of all, thanks for all the great replies on NetPro. I'm somewhat of a newbie on ACE, so they definitely come in handy.

In terms of resource allocation, when configuring your resource-classes, are there any guidelines, best practices etc.? I know that it's recommended by Cisco not to let syslog gain maximum resource, but to restrict it.

thanks

/Ulrich

sumaiyausa Wed, 10/28/2009 - 02:10

Hi Gilles,

I am not able to access the ACE module from the switch. Can you please assist me.

C6509#session slot 2 processor 0

Error: context name (id:0) cannot be determined.

What can i do now???

Regards,

Sum

Gilles Dufour Wed, 10/28/2009 - 03:45

You can try the console.

If that does not work, you won't have any other choice but to reboot.

You should upgrade to A2(1.6a) to get the recent fixes in this area.

Like:

CSCsv98101 ACE blade failed to allow any logins and indicated sysmgr respawning

Gilles.

sumaiyausa Wed, 10/28/2009 - 05:15

Hi Giles,

Thanks for your response. I am now able to login to ACE though switch but I am not able to run anything on the ACE.

1) When I run the cmd:

ACE/Admin# show users

Error: AAA tnrpc call failed to get context name

2)ACE/Admin#wr mem

Generating configuration....

3)ACE/Admin#show tech-support

show tech cmd output nothing displays blankscreen....

Kindly let me know whts happening to my ACE its under production.. Thanks in advance.

Regards,

Sum

Gilles Dufour Wed, 10/28/2009 - 09:33

Sum,

this is the same known issue that prevented you to access it remotely as well.

There were many ddts related to these problems and they all have been fixed in version A2(1.6a).

There is no workaround except to reboot the device.

In the current state, the box should still be able to process traffic but most of commands will fail.

G.

Gilles Dufour Wed, 10/28/2009 - 04:01

Ulrich,

I'm glad you do appreciate the answers you're getting on this forum.

To answer your question, I would say you need to pay attention to some resources.

1/ management connection.

you don't want to set the min value too low ... otherwise you risk loosing access to the box or having the box not being able to send ping or arp request.

I would say try to set the minimum to 5%

2/ sticky resource.

Here the resource does not move between min and max like the others. It stays at the minimum value and if your sticky table is full, we will reuse an old entry.

So make sure to set a realistic minimum.... I would say at least 20% to being with and then increase it if necessary.

In general the most common mistake I see is that people tends to set a very very low minimum like 0.01%.

This usually leads to problem when one context start taking all the resources.

You should think about how many context you expect. Then divide 80% (not the full 100%) by the number of context and use this value as the default minimum.

Then monitor for any denies and increase/decrease where necessary.

Gilles.

sumaiyausa Wed, 10/28/2009 - 06:52

Giles,

Is this issue which I mentioned above is due to software?

Software running is A2(1.0)?

Thanks in advance.

Gilles Dufour Wed, 10/28/2009 - 09:07

yes this is due to software.

This is why I provided you with the ddts number and suggested to upgrade to A2(1.6a).

Gilles.

sumaiyausa Thu, 10/29/2009 - 02:42

Hi Giles,

Can I upgrade my ACE module directly to A2(3.0)from my current version which is A2(1.0)??? Is this compatible ??

Also I assume that A2(3.0) has all bug fixes of A2 (1.X) release am i right???

Thanks in advance.

Regards,

Sum

Gilles Dufour Thu, 10/29/2009 - 09:10

A2(3.0) is not an ace module version but an ACE appliance software.

They are not compatible.

The latest version for the module is A2(1.6a)

Gilles.

Gilles Dufour Thu, 10/29/2009 - 09:44

sorry...my mistake.

This is indeed the new software for the ace module ...so new I wasn't ready to see you referencing it :-)

So, yes this is fine for the module and you can upgrade directly without passing by another version.

Thanks,

Gilles.

JeramyKoval Thu, 11/05/2009 - 08:43

There is a A2(3.0) version available from the download section. If this is only for the appliance then the site needs fixed.

c6ace-t1k9-mz.A2_3_0.bin

Release Date: 12/Oct/2009

Catalyst 6500 Application Control Software for ACE Service Module

Size: 30786.49 KB (31525359 bytes)

Gilles Dufour Thu, 11/05/2009 - 23:53

this one is for the module.

I correct myself in a previous post.

Everything A2 is for the module.

I misread the version and thought it was A3... which is for the appliance.

Gilles.

Gilles Dufour Thu, 10/29/2009 - 01:59

do you want to catch any url that ends with a dot or just the one you showed ?

This is easy to catch the url with the dot, but the redirect is either a fix url or a url containing the old url. We can't reuse just part of the old url.

class-map type http loadbalance url-dot

match http url /support/index[.]

Then define the redirect rserver

rserver redirect HTTP-REDIRECT

webhost-redirection http://%h/support/index

inservice

serverfarm redirect SF_REDIRECT

rserver HTTP-REDIRECT

inservice

You can then tie everything together in a policy-map

policy-map type http loadbalance first HTTP

class url-dot

serverfarm SF_REDIRECT

class class-default

serverfarm ...

But if you need to catch any url and remove the dot, this is not possible

Gilles.

GIULIO FAINI Thu, 10/29/2009 - 07:00

Hi Gilles :-) !!! ,

in ACE, is it possible to make a L7 rule based on the domain-name of a HTTPS GET request???

If I have to enter the domain name like https://Sip.nestle.com , is it enough i use Sip\.nestle\.com in the commnad below "match http header Host"?

For example:

!

class-map match-any L7EDGE

2 match virtual-address VIP tcp eq https

!

class-map type http loadbalance match-all L7CLASS-ACCESS

2 match http header Host header-value Sip\.nestle\.com

class-map type http loadbalance match-all L7CLASS-WEB

2 match http header Host header-value Web\.nestle\.com

class-map type http loadbalance match-all L7CLASS-AV

2 match http header Host header-value AVE\.nestle\.com

!

policy-map type loadbalance first-match L7OCS443

class L7CLASS-ACCESS

sticky-serverfarm ACCESS_STICKY

class L7CLASS-WEB

sticky-serverfarm WEB_STICKY

class L7CLASS-AV

sticky-serverfarm AV_STICKY

!

policy-map multi-match POLICY

...

class L7EDGE

loadbalance vip inservice

loadbalance policy L7OCS443

loadbalance vip icmp-reply active

connection advanced-options TIMEOUT

appl-parameter http advanced-options HTTP_PARAM_CASE

nat dynamic XXX vlan 503

.....

Gilles Dufour Thu, 10/29/2009 - 09:15

The great Mr Giulio Faini... hope you're doing well my friend.

To answer your question, no this is not enough.

Since this is HTTPS the hostname is encrypted like everything else.

So you need to terminate the SSL connection and therefore configure an ssl-proxy server under class L7EDGE.

The certificate will need to be a wildcard certificate - not that ACE does matter - if you don't want users to receiving warning about certificate not matching hostname.

Gilles.

GIULIO FAINI Thu, 10/29/2009 - 22:46

Ciao Gilles !!!

Thanks for the reply. Your posts rock like no others, I follow them with much interests!

One last question about wildcard certificate, does it have to be derived from the real-server certificate or any wildcard certificate is ok?

PS: This ACE is really powerful!

Gilles Dufour Fri, 10/30/2009 - 02:48

it can be any wildcard certificate representing your site ie: *.mycompany.com

G.

UHansen1976 Thu, 10/29/2009 - 08:45

Hi Gilles,

I find myself with a problem, that I can't find the solution for. I get a lot of complaints from our webserver department, that many users experience a "Page cannot be displayed"-error, when accessing our internal portals via ACE. There's no apparent pattern, nothing specific that triggers the problem and it happens completely random (so they say). I've done extensive tracing for the last two days, but have been unable to either find something specific or reproduce the problem myself. The webserver only logs several HTTP-400 msg.

This problem is absent, when users connect directly the webserver.

I've attached my configuration, if it in anyway helps.

Thanks

/Ulrich

Attachment: 
Gilles Dufour Thu, 10/29/2009 - 09:36

what is your software version ?

Did you catch one of the error with your sniffer trace ?

The "page cannot be displayed" is probably a timeout or a RST...but not an error message from the server itself.

We will really need to catch the error to understand it.

Gilles.

UHansen1976 Thu, 10/29/2009 - 13:19

Hi Gilles,

I ran a trace while one of my users interacted with the webserver through ACE. We noted 2 incidents and the corresponding frames in the capture only showed a RST,ACK with the ACE as source-address.

I followed a tcp-stream and nothing appeared out of the ordinary. Handshake, followed by a number of Http-gets and acks, nothing alarming. But for some reason, the stream ends with a reset originating from ACE.

But am I correct in assuming, that the ACE itself does not issue resets, it only forwards whatever the client or the server sends?

I'm currently running 1.4a.

/Ulrich

Gilles Dufour Fri, 10/30/2009 - 02:51

Ulrich,

the ACE coul also generate a RST if it detects a problem with the connection, or if the connection goes beyond the idle timeout.

And the RST is indeed what is causing the user to see the 'page cannot be displayed'.

In your trace, do you also see the server side traffic ?

We would need both to understand what is going on.

You should span the tengig interface so we can see both client side and server side.

Gilles.

UHansen1976 Fri, 10/30/2009 - 06:04

Hi Gilles,

Yes. I've traced on both the client and the serverside. The traces unfortunately reveal nothing particular interesting other than an occasional RST, mostly issued by the client. Every now and then, we see a RST from the webserver. But what's more interesting is, that we've been going through the webserver-log and in there, wee see an abundance of entries which, in short says, that a HTTP200-reply from the applicationserver is somehow logged as a HTTP400. And if this is forwarded to the client, that would explain all the "Page cannot be displayed"-errors. So come monday, we'll do a new trace and focus on this. But evidently, this problem has been going on since this summer, when we upgraded the Webserver software and no one seemed to be bothered by it. The main difference between then and now is, that we've replaced CSS with ACE.

Just to be sure - The 'expect status xxx xxx' only apply to the probes, correct? Theese settings bear no impact on http-traffic recieved by ACE during a client/server flow. I've configured my probes to only accept 200 as a valid response.

Have a good weekend

/Ulrich

Gilles Dufour Fri, 10/30/2009 - 06:29

Ulrich,

again, "page cannot be displayed" is displaued only when there is no response or a RST from the server.

Any 4xx HTTP error code result in the error page to be sent the client which will just display it....For example 403 Forbidden.

The 4xx messages could be the result of stickyness issue but it doesn't explain the page cannot be displayed.

You won't see the problem in the server log since the server didn't display any page.

This can only be troubleshooted by capturing sniffer traces and trying to locate RST or no response from server/ACE

Gilles.

UHansen1976 Mon, 11/02/2009 - 03:08

Hi Gilles,

We've seem to have located the problem. There appears to be some sort of miscommunication between the Websphere and the Webserver. Almost looks, as if the session is terminated before the full payload have been delivered. But everytime, it's the client that sends a FIN,ACK and the weberver (on behalf of the websphere) replies with ACK and then a splitsecond later, the Websphere tries to transmit yet another packet using the same session, upon which the webserver issues a RST. This last part of the transmission is never visible in the trace running in front the ACE. We've determined, that this occurs quite often and has done so for at least 2 months, even prior to the ACE migration, but still, the problem only arise when traffic is forwarded through the ACE. But for now, I'll persue the before mentioned problem. I thank you for your reponses.

/Ulrich

Gilles Dufour Fri, 10/30/2009 - 02:54

The ACE itself is not really aware of what is going on outside itself.

It's the catalyst that will manage everything.

So, VRF or no VRF is not something that changes ACE behavior.

I haven't tested this config myself, but have seen it deployed at some customer locations.

So this should work but if you want to be 100% sure, you should check with your local Cisco reseller or Cisco Sales Engineer.

Thanks.

Gilles.

ROBERTO TACCON Mon, 11/02/2009 - 12:53

Thanks for the docs !

As I need "2Gbps of performance" with RTMP protocol (have not yet indicated the "Maximum connections per second" and the "Concurrent connections") are there any docs about the performance/throughput for the Adobe Real Time Messaging Protocol (RTMP) ?

As indicated by the following doc the CSS 11503 deliver 1.6 Gbps for slot (the ACE-4710 Throughput is 0.5, 1, 2, or 4 Gbps): can I consider the 1.6 Gbps true ?

http://www.cisco.com/en/US/products/hw/contnetw/ps792/products_white_paper09186a0080136856.shtml

The Cisco CSS 11503 makes use of a nonblocking switching fabric, which delivers 20-Gbps throughput to the system. Each module in the system has two 1.6-Gbps connections to the fabric for a total of 3.2 Gbps of full duplex bandwidth to each slot. The fabric supports four levels of priority.

Gilles Dufour Tue, 11/03/2009 - 01:15

I can only recommend you to test the feature you need in a lab.

1.6 Gbps is probably true with the config used at the time of the test.

That does not mean your protocol will also achieve 1.6Gbps.

Gilles.

Marvin Rhoads Sun, 11/01/2009 - 18:41

Thanks Gilles and Cisco for this excellent forum.

Gilles could you please outline the ACE module software release trains or point us to a document that does so? I am looking to upgrade my FT ACE modules from their current A2(1.0a) release to a current one. Would I be advised to go with A2(1.6a), A2(2.1), or A2(3.0) release?

My motivation for upgrading is to remain at a relatively current and patched revision level. I had one ticket on these modules was advised to upgrade as a routine matter as opposed to identifying any resolved caveats for my issue.

My host Catalysts are 6509-E's running 12.2(33)SXH1 on Sup720-10GE's.

Gilles Dufour Sun, 11/01/2009 - 22:53

A2(1.x) is the main release with no new features addition

A2(2.x) was a temporary release where we added a special feature design to handle large configurations.

A2(3.x) is a new release where we added a lot of new features.

There won't be anymore A2(1.x) or A2(2.x) release except maybe an A2(1.6b).

So, if you just want to have a more stable release, go to A2(1.6a).

If you want to have the latest version to stay in synch with the new development you should go to A2(3.0).

Regards,

Gilles.

pengfang Sun, 11/01/2009 - 20:03

Hi Gilles, I couldn't find any documents well explained how layer2 one-arm mode (Asymmetric Server Normalization)works, why we need configure loopback ip address on the server as vip? Is it true vip will be resolved to all backend server MAC addresses in arp table? What the limitation and benefit for this application mode? Thanks.

Gilles Dufour Sun, 11/01/2009 - 22:57

In order to due loadbalancing and asymetric response, you indeed need to have the servers L2 adjacent and you need to configure the Virtual ip on a loopback on the server.

The reason for that is simple.

The clients communicate with the virtual ip.

So they expect a response from that ip and not another one.

If the server response bypasses the loadbalancer, there is no nating possible....so the server needs to use the virtual ip to send the response.

And the only way to do so is have the virtual ip configured on the server.

The loadbalancer will not have an arp entry for the virtual ip.

Instead it will work in transparent mode.

In other words, it will forward the traffic to the server mac-address without nating the destination ip address.

Gilles.

nehakulsum Tue, 11/03/2009 - 00:38

Hi Giles,

I required your support on this:-

am not able to upload the firmware on the ACE module.

Laptop--consolecable---ACE module

rommon >

Loading disk0:c6ace-t1k9-mz.A2_1_6.bin. Please wait ....

file(disk0:c6ace-t1k9-mz.A2_1_6.bin)

open error

loadprog: error - on file open

boot: cannot load "disk0:c6ace-t1k9-mz.A2_1_6.bin"

rommon 1 >

Tried both connecting by Console and though the switch but no luck.

Kindly let me know why is this error and what needs to be done. Is this an indication of Hardware???

I have tried both the way to copy the image i.e though console and though sup module but both fails.

Thanks in advance.

Regards,

Neha.

Actions

This Discussion