Cisco ACE troubleshooting Server performance issue

vinovinom · ‎07-22-2012

Hi guys

We are in process of troubleshooting Server Performance issue through cisco ace.. We are load testing the application and we get performance degradation only when we test for like 1000 - 2000 users. Results are good when tested against 400 users. Is there any avenues that I can start my troubleshooting against? My scenario is below

Load tester ---- > Cisco ACE Appliance 4710 (Routed/PAT) ---->Proxy Server -------> Checkpoint FW ------> Fortigate FW -----> Cisco ACE Appliance 4710 -----> App Server-----> fortigate ----> checkpoint ------> cisco fwsm module ------> DB Servers.

When we run apache bench from proxy directly to app server static page results are good for 2000 users.

but when we run load test from load tester we get performance issues for 2000 users, 400 users is fine.

I see drops in show service policy detail in top ace appliance and bottom ace appliance which indicates that the server is not responding properly.

I have enabled TCP Reuse, selective ACK in first LB and TCP Reuse in second LB which proved to improve some performance.

Your input into this would be much appreciated.

If there is any specific questions that you are interested in asking to help me out please feel free to reply back.

Thanks

Vino

sachinga.hcl · ‎07-24-2012

Please share the output of the following commands so that I can estimate the actual performance parameters situation on your appliance.

Please see more details of these commands as follows"

Overview of Troubleshooting Performance Issues

Before you begin to troubleshoot ACE performance issues, check and

record the following items:

1. Be sure that the correct licenses are installed in your ACE.

2. Record the number of flows that you are sending to the ACE.

3. Record the performance of a single flow.

4. Identify the type of traffic: unidirectional (UDP, management) or

bidirectional (TCP, HTTP, SSL, and so on)

5. Identify the ACE context that is receiving the traffic.

6. Enter the following Exec mode commands and save the output to a file:

* *clear stats all*

* *show clock*

* *show tech-support*

* *show clock*

7. Be familiar with your application setup.

Troubleshooting Performance Issues

To troubleshoot performance issues with your ACE, follow these steps:

1. Display the resources allocated to each resource class in the ACE by

entering the following command:

ACE_module5/Admin# *show resource allocation*

---------------------------------------------------------------------------

Parameter Min Max Class

---------------------------------------------------------------------------

acl-memory 0.00% 100.00% default

0.00% 100.00% RC1

syslog buffer 0.00% 100.00% default

0.00% 100.00% RC1

conc-connections 0.00% 100.00% default

0.00% 100.00% RC1

mgmt-connections 0.00% 100.00% default

0.00% 100.00% RC1

proxy-connections 0.00% 100.00% default

0.00% 100.00% RC1

bandwidth 0.00% 100.00% default

0.00% 100.00% RC1

connection rate 0.00% 100.00% default

0.00% 100.00% RC1

inspect-conn rate 0.00% 100.00% default

0.00% 100.00% RC1

syslog rate 0.00% 100.00% default

0.00% 100.00% RC1

regexp 0.00% 100.00% default

0.00% 100.00% RC1

sticky 0.00% 100.00% default

5.00% 5.00% RC1

xlates 0.00% 100.00% default

0.00% 100.00% RC1

ssl-connections rate 0.00% 100.00% default

0.00% 100.00% RC1

mgmt-traffic rate 0.00% 100.00% default

0.00% 100.00% RC1

mac-miss rate 0.00% 100.00% default

0.00% 100.00% RC1

throughput 0.00% 100.00% default

0.00% 100.00% RC1

2. Display the resources allocated to the context in question by

entering the following command:

ACE_module5/Admin# *show resource usage context C1*

Allocation

Resource Current Peak Min Max Denied

-------------------------------------------------------------------------------

Context: C1

conc-connections 0 0 0 8000000 0

mgmt-connections 0 0 0 100000 0

proxy-connections 0 0 0 1048574 0

xlates 0 0 0 1048574 0

bandwidth 0 0 0 625000000 0

throughput 0 0 0 500000000 0

*mgmt-traffic rate 0 0 0 125000000 0* <------- 1 GBps bandwidth reserved for management traffic

connection rate 0 0 0 1000000 0

ssl-connections rate 0 0 0 5000 0

mac-miss rate 0 0 0 2000 0

inspect-conn rate 0 0 0 6000 0

acl-memory 0 0 0 78610432 0

sticky 0 0 209714 0 0

regexp 0 0 0 1048576 0

syslog buffer 0 0 0 4194304 0

syslog rate 0 0 0 100000 0

*Note*: All bandwidth values are in

units of bytes per second. To convert to bits per second (bps), multiply

the displayed bandwidth value by eight. The ACE reserves 1 Gbps of

bandwidth for management (to-the-ACE) traffic.

3. From the supervisor CLI(for ace modules only), check the connectivity to the back plane by

entering the following command:

cat6k# *show fabric status*

slot channel speed module fabric

status status

2 0 8G OK OK

3 0 8G OK OK

4 0 8G OK OK

*5 0 8G OK OK* *<-------Shows 8 Gbps connectivity to the chassis back plane*

6 0 20G OK OK

8 0 8G OK OK

4. Check the fabric utilization(for ace modules only) by entering the following command:

cat6k# *show fabric utilization*

slot channel speed Ingress % Egress %

2 0 8G 3 2

3 0 8G 0 0

4 0 8G 0 0

*5 0 8G 0 0*

6 0 20G 0 0

8 0 8G 2 3

5. Display the load of the network processors (NPs) in terms of packets

and connection processing for each microengine (ME) by entering the

following command (for ace modules only):

ACE_module5/Admin# *show np 1 me-stats -cpu*

0 proxies open.

ME Utilization Statistics

--------------

RECEIVE: 7

FASTPATH: 44

SLOWTX: 0

TCP_RX: 0

HTTP: 0

IH_RX 0

SSL_ME: 0

CM_CLOSE: 36

X_TO_ME: 0

FIXUP: 0

REASSEMBLY: 0

OCM: 0

TCP_TX: 0

ICM: 39

ACE/Admin# *show np 2 me-stats -cpu*

0 proxies open.

ME Utilization Statistics

--------------

RECEIVE: 9

FASTPATH: 46

SLOWTX: 2

TCP_RX: 0

HTTP: 0

IH_RX 0

SSL_ME: 0

CM_CLOSE: 43

X_TO_ME: 0

FIXUP: 0

REASSEMBLY: 0

OCM: 0

TCP_TX: 0

ICM: 46

*Note*: All *show np* commands must be

entered for both NP1 and NP2 to obtain the total combined results. NPs

operate safely at any percentage of utilization. As ME functions within

the NPs approach 100 percent, the traffic load is stressing the system

close to its architectural limits. Any ME function that reaches 100

percent utilization can cause back pressure and lead to dropped packets

or dropped connections.

6. Monitor the CDE queues and ensure that the Fifo Full drop count

counter is not incrementing by entering the following command(for ace modules only):

ACE_module5/Admin# *show cde health | include Fifo*

Fifo Full drop count 0

Backpressure is the mechanism that the ACE uses to slow the system down

if queues start to fill up internally. Queues that can be affected and

create backpressure are as follows:

* FIFOs for the CDE, NPs, and the Crypto Module

* Internal queues for each ME

It is possible that some packets that are received by the ACE could be

dropped internally if backpressure is applied.

7. Monitor the Fastpath micro engine queues and ensure that the FastQ

Transmit Backpressure, the SlowQ Transmit Backpressure, the Drop:

Transmit Backpressure, and the Drop: Next-Hop queue full counters are

not incrementing by entering the following command:

ACE_module5/Admin# *show np 1 me-stats "-s fp" | include Backpressure*

FastQ Transmit Backpressure: 0

SlowQ Transmit Backpressure: 0

Drop: Transmit Backpressure: 0

ACE/Admin# *show np 1 me-stats "-s fp" | include queue*

Drop: Next-Hop queue full: 0

8. Monitor the TCP micro engine queues and ensure the Drops due to

FastTX queue full, Drops due to Fastpath queue full, Drops due to HTTP

queue full, Drops due to SSL queue full, Drops due to AI queue full, and

Drops due to Fixup queue full are not incrementing by entering the

following command. If TCP receives backpressure, it can drop packets,

fail to ACK packets, and fail to properly track the next packet in the

TCP connection.

ACE/Admin# *show np 1 me-stats "-s tcp" | include queue*

Drop reproxy msg queue full: 0

Drops due to FastTX queue full: 0

Drops due to Fastpath queue full: 0

Drops due to HTTP queue full: 0

Drops due to SSL queue full: 0

Drops due to AI queue full: 0

Drops due to Fixup queue full: 0

The control plane (CP) processor processes all CP traffic (ARP, HSRP,

ICMP to VIPs, routing, syslogs, SNMP, probes, and so on) and handles

configuration management to parse the CLI for syntactical errors and

enforce configuration dependencies and requirements before pushing the

configuration to the data plane.

9. Display a three-way moving average of the CP processor utilization

(updated every five seconds) by entering the following command:

ACE_module5/Admin# *show processes cpu | inc util*

CPU utilization for five seconds: 81%; one minute: 15%; five minutes: 10%

The ACE allocates data-plane memory to guarantee concurrent connection

support for basic Layer 4 connections (such as TCP, UDP, IPsec), Layer 7

connections (proxied flows, typically for application aware load

balancing or inspection, and SSL connection when using SSL

acceleration). The ACE can support the maximum bidirectional concurrent

connection limit regardless of the features enabled.

/*Table 1. Concurrent Connection Support*/

*Connection Type* *ACE Module Limit*

Layer 4 4,000,000

Layer 7 512,000

The state for both directions (client-to-VIP/ACE and server-to-ACE) of a

TCP connection is maintained with distinct connection objects.

10. Display the connection table by entering the following command:

ACE_module5/Admin# *show conn*

total current connections : 6

conn-id np dir proto vlan source destination state

----------+--+---+-----+----+---------------------+---------------------+------+

1 1 in TCP 130 161.44.67.242:2856 10.86.215.134:23 ESTAB

2 1 out TCP 130 10.86.215.134:23 161.44.67.242:2856 ESTAB

4 1 in TCP 130 161.44.67.242:2837 10.86.215.134:23 ESTAB

3 1 out TCP 130 10.86.215.134:23 161.44.67.242:2837 ESTAB

4 2 in TCP 130 161.44.67.242:2857 10.86.215.134:23 ESTAB

3 2 out TCP 130 10.86.215.134:23 161.44.67.242:2857 ESTAB

*Note*: You can add the *detail*

command option to provide the following additional fields: connection

idle time, elapsed time of the connection, byte count, and packet count

for each connection object.

The total current connections counter is also maintained in the output

of the following command:

switch/Admin# *show stats connection*

+------------------------------------------+

+------- Connection statistics ------------+

+------------------------------------------+

Total Connections Created : 124

Total Connections Current : 6

Total Connections Destroyed: 62

Total Connections Timed-out: 58

Total Connections Failed : 0

*Note*: The Total Connections Current

counter counts the number of used connection objects, not the number of

TCP flows. The number of TCP flows can be roughly determined as half the

number of connection objects minus any UDP connections. The Total

Connections Current counter is always up to date and the maximum value

can be 8,000,000.

Because of the Cisco ACE Module’s architecture, with distinct paths for

new and established connections, the number of existing concurrent

connections does not heavily impact the rate at which new connections

can be set up. Nevertheless, a very large number of concurrent

connections will eventually affect the performance of the system in

setting up new connections.

11. Use the command "tcp wan-optimization rtt 0" for slow connections.

The ACE module architecture includes a mechanism where connections can

be moved to the fastpath in order to increase performance for a given

connection. The LB decision is made in the software (proxy) and then

moved to the fastpath (unproxy). In a persistence rebalance scenario,

the proxy/unproxy can occur Many times on a given connection. It is

possible that if a packet enters the system during the transition

Between the proxy and unproxy states, a packet may not be forwarded as

expected and a retransmission may be relied upon. This can affect

performance. As a workaround, it is possible to configure the ACE such

that fastpath forwarding is prohibited This can be accomplished by

configuring a parameter map with the following:

"tcp wan-optimization rtt 0"

HTH

Sachin