01-11-2008 03:14 AM - edited 03-10-2019 03:56 AM
Hello
I apologize if this is not the right forum and appreciate if anyone could point me in the right direction.
I work in a small portuguese university. We have a ASA 5540 + IPS installed here. Unfortunately the guy that installed left without leaving any documentation so I'm a bit lost with this.
We have a 100Mbps FD link to the Internet. The inbound traffic (from the Internet) shows no limit (I can download a DVD iso file at 20Mbps) but the outbound traffic never goes beyond 100-150kbps (from our FTP server, for example or any other servers). We've tried several protocols (HTTP, FTP, SCP) from different servers and the result is always the same.
In between our server farm and the Internet there is only one 3750 switch and the ASA. Every connection is done at 1Gbps (except for the connection to the Internet that is made at 100Mbps FD). If we connect directly to the 3750 we can download things from our server at very high speeds. It's only when we test on the outside that things fail.
Our suspicions is that something is misconfigured on the ASA or the IPS. Can someone please offer a hint?
Thanks in advance for your attention
Solved! Go to Solution.
01-17-2008 03:13 AM
Glad to here that. I'm not positive, but it seems there was a bug related to autonegotiation between 7200 routers and other devices (maybe ASA), not sure which router do you use.
01-11-2008 06:15 AM
First, find out who of them is guilty :) Remove "ips" command from the policy-map configuration on the ASA and test again (this will disable sending traffic to the IPS module. If it doesn't help post at least all the "class-map", "policy-map", "tcp-map", "interface" sections of the ASA config to this forum or firewalling forum. If it helps then reenable sending traffic to the IPS module with the "ips" command, login to the AIP-SSM module with "session 1", do "show stat virtual-sensor clear", repeat tests, do "show stat virtual-sensor" and post the result here. What SW versions are you running?
01-14-2008 07:19 AM
Like I said before my knowledge of this is very basic... I've been trying to do things using ASDM and my simple knowlegde.
I found the policy-map with the ips command:
policy-map my-ips-policy
class my-ips-class
ips promiscuous fail-open
class inspection_default
And I removed it! The result was the speed was a little bit better but still not good enough (in fact I tried an scp over the link and at a point it stalled!)
Since this didn't help, I'm going to post the whole info asked:
class-map my-ips-class
match access-list IPS
class-map OUTSIDE-ftp
match access-list OUTSIDE_mpc
class-map inspection_default
match default-inspection-traffic
class-map http-map1
match access-list http-list2
policy-map type inspect dns preset_dns_map
parameters
message-length maximum 512
policy-map my-ips-class
policy-map http-map1
class http-map1
set connection advanced-options mss-map
class OUTSIDE-ftp
inspect ftp strict
policy-map my-ips-policy
class my-ips-class
ips promiscuous fail-open
class inspection_default
tcp-map mss-map
exceed-mss allow
interface GigabitEthernet0/0
speed 100
duplex full
shutdown
no nameif
security-level 0
no ip address
ipv6 address xxxxxxxxxxxxxxxxxxx
ipv6 enable
!
interface GigabitEthernet0/1
description Ligacao ao Backbone da UE
nameif BKB-UE
security-level 100
ip address xxx.xxx.xxx.xxx 255.255.255.240
ipv6 address xxxxxxxxxxxxxxxxxxxx
ipv6 enable
pim neighbor-filter BKB-UE_multicast
ospf cost 10
!
interface GigabitEthernet0/2
no nameif
no security-level
no ip address
!
interface GigabitEthernet0/2.40
description Rede do Estudio
vlan 40
nameif ESTUDIO
security-level 90
ip address xxx.xxx.xxx.xxx 255.255.255.248
ospf cost 10
ospf network point-to-point non-broadcast
!
interface GigabitEthernet0/2.50
description Rede Guest Utilizadores e-U externos e outros
vlan 50
nameif GUEST
security-level 30
ip address xxx.xxx.xxx.xxx 255.255.255.0
ipv6 address xxxxxxxxxxxxxxxxxx
ipv6 enable
ospf cost 10
ospf network point-to-point non-broadcast
!
interface GigabitEthernet0/2.302
description Rede FWUE
vlan 302
nameif FWUE
security-level 20
ip address xxx.xxx.xxx.xxx 255.255.255.0
ipv6 address xxxxxxxxxxxxxxxxxxxxxx
ipv6 enable
ospf cost 10
ospf network point-to-point non-broadcast
!
interface GigabitEthernet0/3
description Ligacao ao router-world
nameif OUTSIDE
security-level 0
ip address xxx.xxx.xxx.xxxx 255.255.255.240
ipv6 address xxxxxxxxxxxxxxxxxxxxxxxx
ipv6 enable
ipv6 nd suppress-ra
ospf cost 10
!
interface Management0/0
nameif management
security-level 100
ip address 192.168.10.1 255.255.255.0
ospf cost 10
management-only
The software versions are:
Cisco Adaptive Security Appliance Software Version 8.0(3)
Device Manager Version 6.0(3)
Compiled on Tue 06-Nov-07 22:59 by builders
System image file is "disk0:/asa803-k8.bin"
Config file at boot was "startup-config"
router-bkb up 55 days 21 hours
Hardware: ASA5540-K8, 1024 MB RAM, CPU Pentium 4 2000 MHz
Internal ATA Compact Flash, 256MB
BIOS Flash M50FW080 @ 0xffe00000, 1024KB
and
Cisco Intrusion Prevention System, Version 6.0(3)E1
Host:
Realm Keys key1.0
Signature Definition:
Signature Update S307.0 2007-10-16
Virus Update V1.2 2005-11-24
OS Version: 2.4.30-IDS-smp-bigphys
Platform: ASA-SSM-20
Thanks for any help, once again
BTW: can someone please point to something good to read to get up to speed with this.
Thanks
01-14-2008 10:02 AM
1. Please post "service-policy" commands too.
2. What OSes do your servers run?
3. Buy latest Cisco Press book "ASA and PIX firewall..." (not sure will it speed up things or not ;)
4. To put the truth I don't think somebody will be able to troubleshoot this without TAC assistance (I'll try to ask local TAC).
5. Initially I thought about this:
CSCse46220 Bug Details Bug #4 of 12 | < Previous | Next >
ASA: Poor Performance and Out-of-Order packets with SSM module enabled
Symptom:
Out-of-order packets may significantly degrade download performance for certain TCP connections. This is most readily observed in http downloads, but also affects other TCP traffic.
Conditions:
The ASA must be sending traffic to a SSM module. Note that when sending traffic to a SSM module, the ASA will attempt to re-order all packets matched in the access-list of the associated class.
Workaround:
While it does not scale well, one might try removing the problem traffic from inspection by the SSM by adjusting the access-list reference in the class-map.
Increasing the queue-limit under the tcp-map does help with performance, although finding the optimal queue-limit value can take some trial and error to deliver best performance.
One can also try clearing the selective-ack and timestamp options in TCP connections.
An example of adjusting the queue-limit, clearing the selective-ack and timestamp options is shown in the tcp-map below:
tcp-map tmap
tcp-options timestamp clear
tcp-options selective-ack clear
queue-limit <#>
Or this:
Related Bugs
intelligent dynamic queue-limit for tcp normalizer
Symptom: number of buffers are regularly exceed for out of order packets. Conditions: out of order packets We would like an option at the end of queue-limit from the tcp map that invoked an "allocation" process that would dynamically and conservatively up the queue-limit value the algorithm might look something like y= no buffer drops - global buffer drops = number of per conn connection drops x = number of out of order packets y/x = z = % of out of order packets dropped so maybe a while loop if z > 10 % then set queue-limit 4 if z > 15 % set queue-limit 6 up to a queue-limit of 12-13 or whatever values seem to make sense (for percentage and queue-limit)
But it has been fixed in 8.0.
However it might give you an idea. Either your servers send something that ASA don't like (ECNs, wscale, etc.), or there is a problem in "TCP normalization code" in ASA (it is activated when an SSM is present, but probably only if policy-map with "ips" command is present), or the problem isn't related to the ASA at all (ISP / switch / etc. problem).
The SSM module itself shouldn't be the cause of the problem, because it is in the "promisc." mode - packets do not go thru the module, the module receives copies of packets.
01-15-2008 04:04 AM
Hi again,
service-policy my-ips-policy global
service-policy http-map1 interface OUTSIDE
Most of our servers are running linux (2.6.xx kernel version)
I will look for the book and buy it (thanks)
What does TAC mean?
I tried your tcp-map setup and the results are similar. I also disabled the IPS and the "clear local" (like you suggested on other post) and the results are slighty better (100K/s, sometimes)
To avoid ISP possible problems, we have setup a PC with two NICs which is sitting between the ASA and the ISP so that we could do some tests.
The results are identical to the ones I posted before, but one thing that we found out is that if we have several connections on the test machine they all get approximately the same bandwidth (between 15K to 100K).
During one of the tests we had 10 downloads in simultaneous (plus the traffic for the rest of the users) and the outbound traffic never went above 12Mbps.
01-15-2008 05:22 AM
TAC is Technical Assistance Center (Cisco). You need service contract for ASA to call them. They will ask you to capture packets sent by your servers as they appear before the ASA and after the ASA. This is the only way to troubleshoot this.
The only strange thing in your config is:
tcp-map mss-map
exceed-mss allow
class-map http-map1
match access-list http-list2
policy-map http-map1
class http-map1
set connection advanced-options mss-map
I don't know what is in the access-list "http-list2", but anyway, why do you allow packets that are larger than agreed TCP MSS? Do your Linux servers have non-default MTU set? Do you see non-zero counters in "show fragment" on ASA? Do you have throughput problems for Linux servers only? What about non-Unix platforms?
01-17-2008 02:29 AM
Hello again
Those rules relate to one address 62.166.198.202 which misbehaves and just wouldn't be reached from our network.
Meanwhile the problem has been solved: somehow both the asa and the outside router were announcing 100-FD as their negociated speed but they really were not doing that. After a few tests we realized that the link was not ok. As soon as we stopped auto-negotiation on both sides everything started to work. DO you know of any bugs with the ASA software that could cause this?
Anyway I appreciate your help and apologize for your lost time.
I've already looked up the book you mentioned and I'll buy it shortly.
Once again, thanks
01-17-2008 03:13 AM
Glad to here that. I'm not positive, but it seems there was a bug related to autonegotiation between 7200 routers and other devices (maybe ASA), not sure which router do you use.
01-14-2008 11:25 AM
One important thing:
try to remove "ips promiscuous fail-open" once again AND do "clear local" at the ASA prompt. I believe this is needed to activate changes. Then test again.
And, according to the TAC, the Bug CSCse46220 has been fixed in 8.0(3)
Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: