ASK THE EXPERT - CONFIGURING/TROUBLESHOOTING THE APPLICATION CONTROL ENGINE

Unanswered Question
Jun 27th, 2008

Welcome to the Cisco Networking Professionals Ask the Expert conversation. This is an opportunity to learn how to configure and troubleshoot the Application Control Engine with Cisco expert Gilles Dufour. Gilles is a software engineer for the Level 4 to Level 7 switches in the Internet Systems Business Unit since January 2005. He is a CCIE # 3878 in routing, switching and security.

Remember to use the rating system to let Gilles know if you have received an adequate response.

Gilles might not be able to answer each question due to the volume expected during this event. Our moderators will post many of the unanswered questions in other discussion forums shortly after the event. This event lasts through July 11, 2008. Visit this forum often to view responses to your questions and the questions of other community members.

I have this problem too.
0 votes
  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 4.8 (4 ratings)
Loading.
ekonishijunior Fri, 06/27/2008 - 17:50

Hi,

I would like to have a expert opinion about ACE module configured in bridge mode vs routed mode, advantages and disadvantages.

I got some problems in one-arm mode (more complex configs), after all I changed to routed mode.

Gilles Dufour Fri, 06/27/2008 - 23:47

Bridge mode offers the possibility to insert the ACE module transparently between servers and their gateway.

No need for re-addressing or changing servers routing table.

However, it is not always easy to troubleshoot and you need to keep in mind ACE will not source-nat traffic that is bridged.

Router mode will require re-addressing.

But it is easier to troubleshoot and no restriction in terms of source-nating.

Gilles.

Gilles Dufour Sun, 06/29/2008 - 23:38

ACE is a loadbalancer with lots of firewall feature. HTTP inspection can be turned on and OFF depending on how strict you want the blade to be.

However, even if turned off, ACE will still make sure the http header is valid.

One cause of static parse error is the presence of illegal characters in the url (non-ascii character).

Unfortunately there is no way to get more info from ACE itself.

A sniffer trace would be required to analyse it and identify the error.

If you can't find the problem inside the sniffer trace, I would recommened to open a service request with the TAC.

Thanks for your question.

Gilles.

majaj Sat, 06/28/2008 - 21:49

can i install FWSM in routed mode and ACE in bridged mode, so the gateway for servers will be the FWSM.

i tried that , but i can't ping from ACE to FWSM

Gilles Dufour Sun, 06/29/2008 - 23:40

Yes, you can use a design with ACE in bridge mode and FWSM in router mode.

Can you ping from the servers to the FWSM through the ACE ?

When you ping from ACE, do you see a response from the FWSM coming back to ACE if you capture a sniffer trace of the ACE tengig interface ?

Thanks,

Gilles.

followurself Sun, 06/29/2008 - 08:22

Giles,

we are abt to design and implement ACE/FWSM

2 chasis with a module of each.

will it be good idea to use ace/fwsm both in routed mode.

wht failover methods to use for both

want the webserves and database servers to be in 2 diff vlans behind ace.

how the traffic from web server to database and vice versa shd be configured on ace.

there are other traffic types for these web servers and database servers. Like replication and patches. how wd ace cope with that because these traffic shdnt be loadbalance.how to configure?

wht kind of security features shd be enabled on ace for web servers and database servers.

Thanks in advance

Gilles Dufour Sun, 06/29/2008 - 23:51

Hi,

thanks for your question.

Should the traffic between web servers and database servers go through the FWSM ?

If yes, bridge mode might be a better solution for the ACE module.

If yes, and you really want to guarantee that ACE will not *leak* traffic from the database vlan to the webserver vlan, you could also use different contexts.

That would make the config a little bit more difficult.

For the config, anything is possible.

I assume 1 vserver for the webserver and another vserver to the database.

Nothing particular here.

For the rest of the traffic, it really depends on your design. In routed mode, ACE will simply route your traffic from one vlan to the other like any router as long as you permit this traffic inside the access-group access-list.

Gilles.

cscherb Sun, 06/29/2008 - 09:13

Hi Gilles,

is it possible to replace the SSL-certificate the ACE 4710 Device Manager is using ? Out of the box the Device Manager is using a self-signed certificate - but I would like to use a certificate from our internal PKI.

Best Regards

Carsten

AnthonyGZ Sun, 06/29/2008 - 20:36

Hi Gilles

I am using a nat-pool with single ip address (PAT) and it is assigned to a serverfarm for source Natting.

What command can I use on ACE to figure out how many current connections are Natted and How many more connections can be Natted by this Nat-pool.

Thanks

A.

Gilles Dufour Tue, 07/01/2008 - 02:18

That's a good question.

I usually do a 'show np [1|2] me-stat "-socm" | i NAT'

NAT Pool Alloc [addr]: 0 0

NAT Pool Alloc [addr/port]: 0 0

NAT Pool Free [addr]: 0 0

NAT Pool Free [addr/port]: 0 0

If you do pool_alloc_addr_port - free_addr_port you have the currently allocated ports.

The 64000 ports are equally splitted between the 2 IXP. So each get 32k ports.

If you are running out of ports, you should see the following counter incrementing :

NAT Pool Alloc [fail]: 0 0

Another way could be "show xlate | i x.x.x.x | count".

Gilles.

Gilles Dufour Mon, 06/30/2008 - 00:04

Carsten,

technically we could access the linux shell and navigate the directory structure to locate and replace the current cert/key.

You would have to do this after every reboot because there is actually no command to do it.

Gilles.

cscherb Mon, 06/30/2008 - 02:37

OK - thanks a lot for your fast reply. Do you now of any plans to change this ? As you might can imagine it's never a good idea to get used to accepting SSL security warnings.

Gilles Dufour Mon, 06/30/2008 - 06:54

there is no plan currently to change this as far as I can tell.

I'll introduce the idea to the product manager but it might be good for you to ask your Cisco sale/account representative to do the same.

Gilles

aamercado Sun, 06/29/2008 - 22:22

I currently have 2 MS IIS web servers with SSL that I want to move over to the C6K ACE module. Is there a way to export the SSL and import it into the ACE?

Thx

Gilles Dufour Mon, 06/30/2008 - 00:10

The SSL Configuration guide is here

http://www.cisco.com/en/US/docs/interfaces_modules/services_modules/ace/v3.00_A1/configuration/ssl/guide/sslgd.html

And from there you can find the documentation on importing keys and certificates.

http://www.cisco.com/en/US/docs/interfaces_modules/services_modules/ace/v3.00_A1/configuration/ssl/guide/certkeys.html#wp1029280

So, you should be able to export the key/cert from the IIS server into a pkcs12 file.

I believe you then have to split the file in a key and a cert with openssl before you can import everything into the ACE module.

Gilles.

followurself Mon, 06/30/2008 - 02:02

Thanks Giles for your response

is using ACE in one arm mode, running it in a routed mode.

This is what i m planning

1)MSFC--ace in one arm mode---firewall in routed mode---web servers

create a DMZ in fwsm for database servers

do you see any prblms here. MSFC will run ospf to exchange routes over WAN, only reason i m using ace in one arm is because if i need to patch my web servers or database severs, and replcation of database servers, it shall nt use ACE. if you think ACE is fine to be used as bridge , and will nt have any issues for above. i can use ACE is bridge and FWSM in routed mode (dmz for database)

i can only use 2 context on fwsm and i m using the other context for other set of traffic which will nt involve ACE at all.

what i must do for failover of all these components at aggregation layer. do i need to extend vlans or use ISL trunk between MSFC will be a good option

can you suggest on load balancing, sticky and probe methods. we r using oracle 11i. i have gone thru the doc but will need your opinion-Thanks

Gilles Dufour Mon, 06/30/2008 - 07:07

The problem is one-arm topology is that you need to be very careful to guarantee that the servers respond to the client going through the ACE module - no asymetry allowed.

Most people will enable client nat.

This will indeed for the reponse to go back to ACE.

However, this will also prevent your servers to know the client ip address.

All traffic will appear to be coming from a single ip belonging to the ACE module.

Another solution is policy-based routing but not all devices support it and it might be tricky to configure or troubleshoot.

I personally do not recommend one-armed mode unless the amount of traffic generated by the servers and that do not need to be loadbalanced is huge. In this case it would make sense to bypass ACE.

For failover, you usually use another chassis with the same modules and have a portchannel trunk between the 2 chassis.

ACE also requires a dedicated link for the FT traffic.

If your backup site is far away and L2 connectivity is not possible, you could also use Route Health Injection.

For an initial design I always recommend to use the default commands. So, roudrobin should be fine for loadbalancing.

Afte a while running the default, we can check the stats and see - based on your traffic - if another loadbalancing technic should be used.

The best sticky method for HTTP traffic would be cookie insert.

Probes should really be configured depending on your environment and what you consider important to monitor, what you consider acceptable failover time, ...

There is no magic config here.

Gilles.

aamercado Mon, 06/30/2008 - 23:15

I have the following topology and running A2(1.1).

ASA--160---C6k(ACE with Vlan165/177 webfarm)

\160 (app/db serverfarm)

Vlan 160 is my inside serverfarm (ie apps and database)

Vlan 165 is the VIP Vlan

Vlan 177 is the Web servers

160 can ping 177 but cannot rdp

177 cannot ping or rdp into 160

outside clients to 165/177 is fine and 177 can ping to outside and have web access.

How do I troubleshoot or get 177 to access 160 in terms of accessing apps or db servers?

Gilles Dufour Tue, 07/01/2008 - 02:08

Make sure there is no asymetric routing.

Capture sniffer trace in the different vlans and follow the path of the traffic.

If there is asymetry, you can do 'no norm' on all ACE interfaces.

But it would only fix issues related to routing.

If you have loadbalancing issues due to asymetry, it would not help. You will still have to fix the asymetry.

Gilles.

AnthonyGZ Tue, 07/01/2008 - 15:25

Hi Gilles

I have few questions for you

Q1. What is the difference between "ssl probe" and "SSL_PROBE_SCRIPT" script provided with ACE.

Q2. Is SSL_PROBE_SCRIPT in probe: directory any different from the script file available on CCO for 2.1 code.

(I am currently running 1.63 code and planning to upgrade to 2.1.)

Q3. If I upgrade my ACE from 1.63 to 2.1 Will the SSL_PROBE_SCRIPT and other scripts in probe: directory be upgraded as well?

Q4. If I upload a custom Script. Can it be used by multiple contexts? Do I need to upload it once and call it in different Contexts or I need to upload it n times for n contexts using unique names.

I am confused with the following statement in ACE user guide

"The filename that you assign the script must be unique across the contexts. You will use this filename when you load the script into the ACE memory and configure the probe"

Q5. Is there a way to monitor the utilization of NAT-Pools. I am using PAT with a single IP address for source NAT and dont want to drop connections due to absence of NAT resources.

Thanks in advance

Anthony.

Gilles Dufour Tue, 07/01/2008 - 23:31

A1: The ssl probe is a true SSL connection with an HTTP request. The script is just the translation of the CSM script which only does a SSL handshake.

A2: I believe they are the same.

A3: If the script needed to be modified because of known issues, yes they would be upgraded with the image. But I don't know of any issues with the SSL script. So it should be the same.

A4: You will need to copy it to every context.

I personally use the same file name in every context.

A5: I have answered the same question in this topic. Please refer to the previous answer.

Thanks,

Gilles.

AnthonyGZ Wed, 07/02/2008 - 16:37

Thanks for your earlier responses.

I have few more in line of your responses.

Q Is there a way to copy a probe script from "disk0: of Admin context", to "disk0: of any other context". All of my other context are in one arm mode with no access to any ftp/tfp servers.

Q You mentioned 'show np [1|2] me-stat "-socm" | i NAT' command for looking at NAT stats. All "sh np x me-stat" commands outputs two columns. What does second column represents? Delta?

Q. Is there any detailed documnetation available on CCO that explains these me-stat commands?

Thanks

Anthony

Gilles Dufour Wed, 07/02/2008 - 23:38

Anthony,

there is no command to copy files from one context to the other.

You can use the MSFC as a TFTP server.

Copy the file to the MSFC disk and then copy it from there to all the contexts.

The 2nd column is supposed to be the delta but it is acting weirdly right now. I would simply ignore it for now.

Those counters are *not yet* documented on CCO. But the nat counters should be very explicit.

In case you really need to know the exact meaning of a counter, post a question to the forum and I will certainly reply.

Gilles.

AnthonyGZ Thu, 07/03/2008 - 17:11

As per my understanding "persistence rebalance" enables ACE to look at

every GET request in a single TCP connection and select appropriate server farm

as per the L7 criteria defined in LB policy maps.

How does "Persistence rebalance" works with Stickiness?

If I am using Source IP based stickiness and one client gets stuck on the basis of first

Get request, What will happen to the subsequent requests?

Is there any significance of using "Persistence rebalance" with source ip stickiness?

In the similar lines what happens if "Persistence rebalance" and "TCP Reuse" both are configured.

Thanks

Gilles Dufour Fri, 07/04/2008 - 04:43

Persistence rebalance is needed when you have proxy servers connecting to your vip and you do stickyness based on payload information like cookie.

So, if you are using sticky srcip, it makes no sense to enable this option.

With tcp-reusem, persistence rebalance still has the same effect. Every request is treated independently. So for each request we will look if there is an existing tcp connection to the server that we can reuse.

Gilles.

ingredosi Tue, 07/01/2008 - 21:26

Hi, Gilles

We have several questions:

In "cookie insert" persistence:

- Is the path cookie configuration supported?

- How is the policy configured so we can match the inserted cookie and maint persistence?

With CSS, the unbalanced traffic affects CPU consumption.

In the ACE firewall load balancing topology and two contexts (internal//external),

part of the traffic does not get balanced in the internal context for several reasons (e.g., direct traffic to servers)

Is the ACE performance affected?

In CSS it is possible to do SNAT with different IP by service.

We need to do so when the servers are located in different internal/external networks.

Is something similar supported by ACE?

Thanks

JM

Gilles Dufour Tue, 07/01/2008 - 23:11

There is no option to set the path.

The default is "/".

Simply create the sticky group

sticky http-cookie NAME MySticky

cookie insert

serverfarm MyServers

Then inside your http policy do

policy type loadbalance first-match MyPolicy

class class-default

sticky-serverfarm MySticky

You don't need to worry about the value of the cookie. ACE will take care of it for you.

ACE is more complex than the CSS.

There is 2 CPU and 16 MicroEngines.

The CPU will not be affected by routed traffic.

But the microengines will still be involved to switch/route the traffic.

So, there will be an impact but very little.

This routed traffic will also consume your Bandwidth. If you have a license for 4Gbps, make sure this routed traffic does not consume everything.

The nat-pool is selected with the outgoing interface.

So, if traffic is sent to server1, on interface A, you could use nat-pool N1.

If server is sent to server2, on interface B, you can use nat-pool N2.

It used to be the only way to have different nating.

With 2.1.1 you can also specify the nat-pool associated with a serverfarm.

switch/Admin(config)# policy-map type loadbalance first-match SF_Linux1

switch/Admin(config-pmap-lb)# class class-default

switch/Admin(config-pmap-lb-c)# nat ?

dynamic Configure dynamic network address translation

switch/Admin(config-pmap-lb-c)# nat dynamic ?

<1-2147483647> Specify network address-pool for translation

switch/Admin(config-pmap-lb-c)# nat dynamic 1 ?

vlan VLAN interface

switch/Admin(config-pmap-lb-c)# nat dynamic 1 vlan 20 ?

serverfarm Specify serverfarm

switch/Admin(config-pmap-lb-c)# nat dynamic 1 vlan 20 serverfarm ?

backup Choose backup serverfarm for this NAT

primary Choose primary serverfarm for this NAT

switch/Admin(config-pmap-lb-c)# nat dynamic 1 vlan 20 serverfarm primary ?

Carriage return.

switch/Admin(config-pmap-lb-c)# nat dynamic 1 vlan 20 serverfarm primary

Gilles.

htsiartas Tue, 07/01/2008 - 22:09

hi

Can ACE appliance be used a HeadEnd SSLv3 for non HTTP traffic. The current SSL headend is on OpenSSL linux from stunnel.org. can ACE decrypt SSL with certificates (just like HTTP) then sent the unside(cleartext) TCP traffic internally?

so the question is can SSL be used for generic TCP traffic? any limitations to that?

thanks

h

Gilles Dufour Tue, 07/01/2008 - 23:53

FTP over SSL is not supported

SMTP over SSL is not supported

LDAP over SSL is not supported.

The main reason is that these protocols require inspection after decryption.

If you just want to pass the decrypted data to the server, it does work.

Gilles.

g-georgiou Thu, 07/03/2008 - 23:28

Hi Gilles,

Is any generic TCP connection supported? To elaborate, if a client is able to encapsulate a TCP connection into an SSL connection (see stunnel for example) can the ACE decrypt and loadbalance to a server farm?

Thanks,

George

htsiartas Sat, 07/05/2008 - 12:13

"

FTP over SSL is not supported

SMTP over SSL is not supported

LDAP over SSL is not supported.

The main reason is that these protocols require inspection after decryption.

"

I don't understand what the issues is with with FTP, SMTP, LDAP?

also datasheet is confusing since is mentions support for LDAP

http://www.cisco.com/en/US/prod/collateral/contnetw/ps5719/ps7027/Data_Sheet_Cisco_ACE_4710.html

• SSL accelerated protocols: HTTPS, Secure IMAP (IMAPS), Secure Lightweight Directory Access Protocol (LDAPS), Secure Network News Transfer Protocol (NNTPS), Secure POP Version 3 (POP3S), and Secure Telnet (STELNET)

Gilles Dufour Mon, 07/07/2008 - 05:21

The showstopper is FTP uses 3 clear-text FTP messages before starting SSL.

LDAP and SMTP are actually supported.

Thanks for correcting me on that one.

Gilles.

amazumde Thu, 07/03/2008 - 21:46

Hello,

I am using an ACE 20 module in 6500 switch. My configuration is one armed where multiple layer 3 vlans from the switch have been allocated to the ACE. The reason for this configuration is that I didn't want to put server vlans behind the ACE as in this environment, the server and client could be anywhere. Now here is the problem.

I am using a VIP to load balance 2 real servers on ports 80 & 443. The VIP is in a completely different subnet that the servers. The servers use DSR to respond back directly to the clients. Connectivity at port 80 seems to be working fine but not at 443. I can connect to port 443 once, and then when I try again I can't. After 5-10 minutes, I can do it again. I am totally baffled and even TAC is not able to find any innediate reason for this kind of behavior. Two TAC engineers have verified that the configuration is correct. Normalization is turned off on the vlan interface and the farm has been set as transparent.

If I connect directly to the servers everything works fine. The servers are set with loopback interfaces with address of the VIP to support DSR.

Please help.

Gilles Dufour Fri, 07/04/2008 - 04:35

What's the case number ?

It will be easier for me to get the config and all the information already provided.

Do you terminate SSL traffic on the ACE module ?

L7 traffic (including SSL terminated traffic) is also normalised even if normalization is turned off.

Gilles.

amazumde Fri, 07/04/2008 - 07:48

Hello, The case number is 609011151. The SSL traffic is not terminated on the ACEs. The client just sends traffic to the VIP which bounces off the ACE to the real servers which are listening at the VIP address configured on their loopback interface (FreeBSD servers).

Gilles Dufour Sat, 07/05/2008 - 04:09

The TAC is following the right procedure.

Please continue working with them.

The sniffer trace of the tengig interface during the failure is important.

Gilles.

Gilles Dufour Sat, 07/05/2008 - 09:08

This is just a partial config.

Nothing wrong with it, but don't have the full picture.

If this was a basic config issue, you could never connect. You said you can connect 2 times and then it fails for 5 -10 minutes.

This is weird.

We'll need the sniffer trace and and a show tech captured before and after the test.

Gilles.

diro Mon, 07/07/2008 - 05:03

Hi Gilles,

Imagine the following situation I have an ssl proxy server running with a certificate that's going to be invalid in a few weeks.

How would I renew this certificate without interrupting the ssl service?

Thx,

Dimitri

Gilles Dufour Mon, 07/07/2008 - 07:57

ACE2.0 support "Transparent Certificate update".

There is no need for suspending any policy/class.

Make a modification to the ssl-proxy service and once the certificate is pushed to the data plane the new connections will receive the new certificate. There may be an impact to a few connections in the midst of receiving a multi packet cert when the new cert is sent, but this would be a minimal impact.

Gilles.

aamercado Mon, 07/07/2008 - 15:40

3.0(0)A2(1.1)- I have ACE on a C6k with 2 servers in a serverfarm set to default of roundrobin.

On my client, I open multiple browser web pages and it mostly goes to one server becuase for testing, we setup the web page to show the name of the web server so we can identify if roundrobin works.

How do I test to verify roundrobin (or any loadbalancing method works)?

I am using source ip address so what sh/deb commands can I use to find source address?

Thx

Gilles Dufour Tue, 07/08/2008 - 02:09

If you really want to test the loadbalancing methods, you will need a traffic generator that can simulate multiple clients.

With a few connections and a single client, you will see strange results.

One reason is that ACE is actually 2 Network Processors and the traffic is split between the 2 using a hash algorithm.

Each NP will then perform the loadbalancing indepently from the other.

You may also have stickyness configured which would result in all traffic from a client going to a single server.

Gilles.

aamercado Tue, 07/08/2008 - 09:01

Is there a show or debug command to give me some insight?

For sticky, I am using

sticky ip-netmask 255.255.255.255 address source SOURCE-IP-STICKY

timeout 10

serverfarm RBWEB

policy-map type loadbalance first-match LB-VIP-RBWEB

class class-default

sticky-serverfarm SOURCE-IP-STICKY

policy-map multi-match LB-VIP

class VIP-RBWEB-WWW

loadbalance vip inservice

loadbalance policy LB-VIP-RBWEB

loadbalance vip icmp-reply

loadbalance vip advertise active

connection advanced-options IDLE-TIMEOUT

Gilles Dufour Wed, 07/09/2008 - 00:27

There is no debug for loadbalancing decision because this is done at low level in one of the Network Processor.

But there is no need to debug to understand the problem.

You have sticky src ip.

Do a 'sho stick database" and a 'show conn' while executing your test.

The sticky option will force the traffic from a specific client to a single server.

This is the goal of stickyness.

If you want to check roundrobin behavior, remove stickyness and test again.

You will see the difference.

Gilles.

yatao Tue, 07/08/2008 - 10:56

Gilles,

I have a few questions for you regarding ACE 4710 appliance:

1. Which one is prefered for connecting FT interface: crossover cable or using switch ports?

2. What is recommended resource configuration for 1 admin and 2 user contexts? The objective is reserve maximum resource to user contexts. I had issue with syslog when use default max unlimited.

3. What is Cisco recommended way for managing/configure ACE appliance: Web GUI or CLI?

4. Any future enhencement for updating capture with tcpdump?

Thanks,

Yatao

Gilles Dufour Wed, 07/09/2008 - 00:36

Yatao,

A1: I believe a crossover cable would be a better solution. No risk to have switch causing issues.

A2: syslog is the only resource you do not want to be unlimited.

Then there is the sticky resource which doesn't grow. It will only use the minimum even if the maximum is unlimited.

I would personnaly reserve min 20% for each user context with max unlimited except for syslog and sticky (set the max equal-to-min).

This leaves you 60% if you want to add new contexts.

A3: We have no recommendation. I personally prefer the CLI because the day there is a big outage and the GUI is not responsive, you already know the commands.

A4: The capture utility is very limited because this function had to be implemented in the Nework Processs which are not linux machines. So no tcpdump available and no possibility to add it.

Moreover, a NAM blade in the chassis is a very good addition and really helps capturing traces remotely. a VERY VERY useful tool.

If you don't have a NAM I still recommend capturing traces using the Sup monitor commands.

Gilles.

yatao Wed, 07/09/2008 - 05:29

Thanks for your answers. As always, you are the most helpful.

I do want to suggest to add "delay" option for FT preempt failover/failback, just like HSRP. That way flapping can be prevented, and allow time for probe to recover when failback.

Thanks,

Yatao

AnthonyGZ Wed, 07/09/2008 - 15:54

Hi Gilles

I have an ACE module running A2(1.1).

Currently there are no reals connected to it.

When I call scripted probe from admin context I get (which make sense - as there are no reals to respond)

"Internal error: Script was terminated due to time out

Last status code : 30004

"

But When I use the same scripted probe from non-admin context I get

"

Internal error: Script error

Last status code : 30006"

Which made me think that there is some issue with calling Scripted froms from non-admin context.

I looked in bug toolkit and found CSCsg10189.But As per bug details it got fixed in 3.0(0)A2(1.0).

I am runnning A2(1.1).

Is it not fixed in A2(1.1)?

Thanks

Actions

This Discussion