Poor Performance

Unanswered Question
Aug 24th, 2010

I have been working with Pix Firewall and ASA 5550.

I am using the default policy configuration including inspect http.

I got throughput 10 times biger without using inspect http (on both pix and asa) when moving files :

wget http://averybigfile

I have this problem too.
0 votes
  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 0 (0 ratings)
Loading.
terrygwazdosky Tue, 08/24/2010 - 10:57

Could you post the output of the following commands?:

sh run service-policy

sh run class-map

sh run policy-map

Rosa Ladeira Tue, 08/24/2010 - 11:34

on pix firewall:

pix# show running-config policy-map
!
policy-map global_policy
class inspection_default
  inspect dns maximum-length 4096
  inspect ftp
  inspect h323 ras
  inspect netbios
  inspect rtsp
  inspect skinny
  inspect esmtp
  inspect sqlnet
  inspect sunrpc
  inspect tftp
  inspect sip
  inspect xdmcp
  inspect rsh
  inspect icmp
  inspect http

pix# sh running-config class-map
!
class-map inspection_default
match default-inspection-traffic

pix# show running-config policy-map
!
policy-map global_policy
class inspection_default
  inspect dns maximum-length 4096
  inspect ftp
  inspect h323 ras
  inspect netbios
  inspect rtsp
  inspect skinny
  inspect esmtp
  inspect sqlnet
  inspect sunrpc
  inspect tftp
  inspect sip
  inspect xdmcp
  inspect rsh
  inspect icmp
  inspect http
!
pix#

on ASA :

asa# show running-config policy-map

!

policy-map type inspect dns preset_dns_map

parameters

  message-length maximum 512

policy-map global_policy

class inspection_default

  inspect dns preset_dns_map

  inspect ftp

  inspect h323 h225

  inspect h323 ras

  inspect rsh

  inspect rtsp

  inspect esmtp

  inspect sqlnet

  inspect skinny

  inspect sunrpc

  inspect xdmcp

  inspect sip

  inspect netbios

  inspect tftp

  inspect icmp

!

asa# show running-config class-map

!

class-map inspection_default

match default-inspection-traffic

!

asa# show running-config policy-map

!

policy-map type inspect dns preset_dns_map

parameters

  message-length maximum 512

policy-map global_policy

class inspection_default

  inspect dns preset_dns_map

  inspect ftp

  inspect h323 h225

  inspect h323 ras

  inspect rsh

  inspect rtsp

  inspect esmtp

  inspect sqlnet

  inspect skinny

  inspect sunrpc

  inspect xdmcp

  inspect sip

  inspect netbios

  inspect tftp

  inspect icmp

!

terrygwazdosky Tue, 08/24/2010 - 12:20

Everything looks pretty baisc. I didn't see the service-policy, but since everything else is default I'm assuming you are just using a global policy and not  interface specific policies.

A few other questions:

  • Is it just http file transfers that are slow?  How is browsing in general?
  • Are you doing any URL filtering?  I've had occasional issues with Websense that caused slow web traffic.
  • What version of software are the PIX and ASA running?

Also, try this:

  • Run this command which will clear your service-policy statistics: "clear service-policy global" (unless you are using interface specific policies)
  • Enable http inspection with defaults
  • Run this command to outline what the traffic flow matches: "sh service-policy flow tcp host eq 1025 host eq http".  It will most likely just hit the defaults.
  • Perform testing
  • Run this command: "show service-policy inspect http", and look to see if there are any drops or resets that may indicate protocol violations and the like
  • If nothing show up with the above it might be worth setting up a capture for traffic to and from the website you use for testing and then looking at the results in Wireshark to look for wierdness.

Let us know how it goes.

Scott Nishimura Tue, 08/24/2010 - 12:52

Hi Rosa,

Is there a reason you are running the http inspection.. It will do strict http checking so it can slow down the traffic.  The ASA will already be looking at the tcp traffic so its more like double checks that are going on.   If you are transfering data using port 80, then the inspection will definitely be analyzing the traffic.

http://www.cisco.com/en/US/docs/security/asa/asa82/command/reference/i2.html#wp1735782

Alot of people are not running with http inspection unless you need the strict checks that it does.

show perfmon will show you the packets per sec that http is looking at along with tcp fixups, etc.

regards,

scott

Rosa Ladeira Tue, 08/24/2010 - 13:33

Hy Scott, thanks for your answer.

I was not sure that not running inspect http would be a correct choice.

According to your answer, running inspect a protocol will "allways" slow down performance ?

I had configured :

  inspect dns preset_dns_map
  inspect ftp
  inspect h323 h225
  inspect h323 ras
  inspect rsh
  inspect rtsp
  inspect esmtp
  inspect sqlnet
  inspect skinny 
  inspect sunrpc
  inspect xdmcp
  inspect sip 
  inspect netbios
  inspect tftp
  inspect icmp
  inspect http

sip, h family rtsp skiny will cause VoIP degradation  ?

Scott Nishimura Tue, 08/24/2010 - 13:39

Hi Rosa,

its up to you whether you need the extra strict http checks.  A lot of sites do not adhere to standards.  As for it always causing performance problems-- not really, but it does add extra inspection and when you have the firewall doing inspections, it is sent to the cpu for further processing.  So if you are having http file transfer, it can slow down the traffic as it has to look at every packet.

Sip inspection is to open up additional secondary pinhole conns so that is what that inspection is doing and is different from the http which is looking at all port 80 traffic.

regards,

scott

Rosa Ladeira Tue, 08/24/2010 - 13:11

Thanks for helping.

Answering your questions:
* Is it just http file transfers that are slow? 
Yes. Many user had questioned about.

* How is browsing in general?
It is fine.

* Are you doing any URL filtering?
No. I have done ASA's factory reset before testing in order to use only 2 interfaces.
Each ASA's interface has a host. One of them wget's.

* What version of software are the PIX and ASA running?
ASA Version 8.2(1)
Cisco Adaptive Security Appliance Software Version 8.2(1)
Device Manager Version 6.2(3)
Hardware:   ASA5550, 4096 MB RAM, CPU Pentium 4 3000 MHz
Internal ATA Compact Flash, 256MB
BIOS Flash M50FW080 @ 0xffe00000, 1024KB

Encryption hardware device : Cisco ASA-55x0 on-board accelerator (revision 0x0)
                             Boot microcode   : CN1000-MC-BOOT-2.00
                             SSL/IKE microcode: CNLite-MC-SSLm-PLUS-2.03
                             IPSec microcode  : CNlite-MC-IPSECm-MAIN-2.04


* Run this command which will clear your service-policy statistics:
"clear service-policy global" (unless you are using interface specific policies)
Done

* Enable http inspection with defaults
Done

* Run this command to outline what the traffic flow matches:
"sh service-policy flow tcp host eq 1025 host eq http". 
It will most likely just hit the defaults.

show service-policy flow tcp host 147.65.32.25 eq 1025 host 147.65.1.48
eq http

Global policy:
  Service-policy: global_policy
    Class-map: class-default
      Match: any
      Action:
asa#    Output flow:


* Perform testing
Now it runs fine & fast

* Run this command: "show service-policy inspect http", and look to see if there are any drops
or resets that may indicate protocol violations and the like

asa#             show service-policy inspect http

Global policy:
  Service-policy: global_policy
    Class-map: inspection_default
      Inspect: http, packet 1035582, drop 0, reset-drop 0


* If nothing show up with the above it might be worth setting up a capture for traffic to and
from the website you use for testing and then looking at the results in Wireshark to look for wierdness.

Let us know how it goes.
Rosa Ladeira Wed, 08/25/2010 - 07:20

Hi Terry.

Yesterday I have made tests using a 100Mbs network interface client host.

Results were mascarade.

Today I have used a 1000Mbs network interface's client.

I have done ASA's factory reset before testing.

Following your suggested configuration on ASA, I got:

global policy with no inspect http : throughput -> 450Mbs

global policy with inspect http : throughput -> 200Mbs

Moving client host to http server subnet (no ASA between them) throughput scales to 900Mbs.

As you can see above Scott has suggested not using inspect http.

What do you think about ?

Jitendriya Athavale Wed, 08/25/2010 - 07:45

i belive this could be because of out of order packets...

can u please apply captures on outside and inside and see if you see any out of order packets

terrygwazdosky Wed, 08/25/2010 - 07:48

Wow, that is a big difference.  I don't notice anything approaching that level of slow down with 1000+ users on an ASA 5520 and that's with http inspeection and Websense filtering.

If you don't need the http filtering, at least in the short term, you may want to leave it off for until you can get to the bottom of this issue.  I'm wondering if maybe you've hit a bug with the version of code you have.  It might be worth opening a TAC case to get some further assistance with troubleshooting.

I'd also reccommend setting up a capture of the traffic and reviewing the results in Wireshark or whichever program you use for packet analysis.

Kureli Sankar Wed, 08/25/2010 - 07:58

Enabling http inspection expects the packets to arrive in order (so we can inspect). If they don't arrive in order then, the ASA has to hold them until all the packets arrive.  The hold buffer or queue is very small so, there are chances that the packets may be just dropped.  Packets arriving out of order is the nature of the internet and may be you can reach out the ISP and ask them why we see (if you really see out of order packet via captures) out of order packets and ask if they can do anything about this.

Http inspection also sends syslogs about the URL requested by each host on the inside.

So, leave http inspection turned off unless there is a requirement that you have to have that on due to some Sarbanes Oxley regulation or some thing of that nature.

https://supportforums.cisco.com/docs/DOC-8982#http_inspection_enabled

-KS

Actions

This Discussion