Re: cisco asa 5580-20 limit ?

inteltechs · ‎06-16-2010

Our cisco 5580 is very slow. Wonder if we hit the limit or some sort of bugs. Can anyone help?

show perfmon

PERFMON STATS: Current Average

Xlates 91/s 0/s

Connections 39631/s 0/s

TCP Conns 39602/s 1/s

UDP Conns 14/s 1/s

URL Access 0/s 0/s

URL Server Req 0/s 0/s

TCP Fixup 0/s 0/s

TCP Intercept Established Conns 0/s 0/s

TCP Intercept Attempts 0/s 0/s

TCP Embryonic Conns Timeout 1727/s 0/s

HTTP Fixup 0/s 0/s

FTP Fixup 0/s 1/s

AAA Authen 0/s 0/s

AAA Author 0/s 0/s

AAA Account 0/s 0/s

VALID CONNS RATE in TCP INTERCEPT: Current Average

N/A 92.00%

show cpu

CPU utilization for 5 seconds = 62%; 1 minute: 62%; 5 minutes: 60%

show processes cpu-usage

PC Thread 5Sec 1Min 5Min Process

080581ac 2b770a18 0.0% 0.0% 0.0% block_diag

0806bd35 2b770040 0.0% 0.0% 0.0% Reload Control Thread

08073ec6 2b76fe48 0.0% 0.0% 0.0% aaa

080a5c76 2b76fa58 0.0% 0.0% 0.0% CMGR Server Process

080a6185 2b76f860 0.0% 0.0% 0.0% CMGR Timer Process

081d592c 2b76f080 0.0% 0.0% 0.0% dbgtrace

0847f82c 2b76e6a8 0.0% 0.0% 0.0% I/O Bridge

0854157f 2b76dec8 0.0% 0.0% 0.0% IPMI Poll Thread

08cb326d 2b76d4f0 0.0% 0.0% 0.0% netfs_thread_init

092d6f95 2b76c920 0.0% 0.0% 0.0% Chunk Manager

0891f36e 2b76c728 0.0% 0.0% 0.0% PIX Garbage Collector

08912774 2b76c530 0.0% 0.0% 0.0% IP Address Assign

08ad4ad6 2b76c338 0.0% 0.0% 0.0% QoS Support Module

0898806f 2b76c140 0.0% 0.0% 0.0% Client Update Task

093234ba 2b76bf48 0.0% 0.0% 0.0% Checkheaps

08af82a5 2b76b960 0.0% 0.0% 0.0% Quack process

08b4eb02 2b76b768 0.0% 0.0% 0.0% Session Manager

08c5f035 2b76b378 0.0% 0.0% 0.0% uauth

08bff475 2b76b180 0.0% 0.0% 0.0% Uauth_Proxy

08c36985 2b76ab98 0.0% 0.0% 0.0% SSL

08c5d496 2b76a9a0 0.0% 0.0% 0.0% SMTP

08c571b6 2b76a7a8 0.0% 0.0% 0.0% Logger

08c51afe 2b76a5b0 0.0% 0.0% 0.0% Thread Logger

08e3f1c2 2b7697e8 0.0% 0.0% 0.0% vpnlb_thread

082a42cd 2b769200 0.0% 0.0% 0.0% TLS Proxy Inspector

08b68313 2b769008 0.0% 0.0% 0.0% emweb/cifs_timer

086a3e07 2b768e10 0.0% 0.0% 0.0% netfs_mount_handler

08532468 2b768c18 0.0% 0.0% 0.0% arp_timer

0853bd0c 2b768a20 0.0% 0.0% 0.0% arp_forward_thread

085b0fa5 2b768828 0.0% 0.0% 0.0% Lic TMR

08c62611 2b768630 0.0% 0.0% 0.0% tcp_fast

08c657f0 2b768438 0.0% 0.0% 0.0% tcp_slow

08c8f719 2b768240 0.0% 0.0% 0.0% udp_timer

08105248 2b767e50 0.0% 0.0% 0.0% CTCP Timer process

08deefb3 2b767c58 0.0% 0.0% 0.0% L2TP data daemon

08defd83 2b767a60 0.0% 0.0% 0.0% L2TP mgmt daemon

08ddc098 2b767868 0.0% 0.0% 0.0% ppp_timer_thread

08e3f677 2b767670 0.0% 0.0% 0.0% vpnlb_timer_thread

081235f7 2b767478 0.0% 0.0% 0.0% IPsec message handler

081343ac 2b767280 0.0% 0.0% 0.0% CTM message handler

089e68f9 2b767088 0.0% 0.0% 0.0% NAT security-level reconfiguration

08b237b8 2b766e90 0.0% 0.0% 0.0% ICMP event handler

08db015f 2b766c98 0.0% 0.0% 0.0% Dynamic Filter VC Housekeeper

08876883 2b766aa0 0.0% 0.0% 0.0% IP Background

081aa254 2b7668a8 0.0% 0.0% 0.0% tmatch compile thread

08a13555 2b7666b0 0.0% 0.0% 0.0% Crypto PKI RECV

08a1617a 2b7664b8 0.0% 0.0% 0.0% Crypto CA

08930e58 2b7662c0 0.0% 0.0% 0.0% uauth_urlb clean

089186df 2b7660c8 0.0% 0.0% 0.0% pm_timer_thread

084c3d15 2b765ed0 0.0% 0.0% 0.0% IKE Timekeeper

084b75cb 2b765cd8 0.0% 0.0% 0.0% IKE Daemon

08c1341a 2b765ae0 0.0% 0.0% 0.0% RADIUS Proxy Event Daemon

08be093b 2b7658e8 0.0% 0.0% 0.0% RADIUS Proxy Listener

08c12017 2b7656f0 0.0% 0.0% 0.0% RADIUS Proxy Time Keeper

0852329c 2b7654f8 0.0% 0.0% 0.0% Integrity FW Task

082ad316 2b765300 0.0% 0.0% 0.0% CP Processing

082ad502 2b765108 0.0% 0.0% 0.0% CP Midpath Processing

082ad12f 2b764f10 0.0% 0.0% 0.0% SRTP Processing

081ef71b 2b764d18 0.0% 0.0% 0.0% ci/console

08411ea8 2b764b20 0.0% 0.0% 0.0% fover_thread

08dc40e5 2b764928 0.0% 0.0% 0.0% lu_ctl

0895149c 2b764730 0.0% 0.0% 0.0% update_cpu_usage

0894aabc 2b764538 0.0% 0.0% 0.0% health_check

0894c68a 2b764148 0.0% 0.0% 0.0% NIC status poll

08405d2c 2b763b60 0.0% 0.0% 0.0% fover_rx

08402440 2b763968 0.0% 0.0% 0.0% fover_tx

0840f40b 2b763770 0.0% 0.0% 0.0% fover_ip

08417431 2b763578 0.0% 0.0% 0.0% fover_rep

0840ed01 2b763380 0.0% 0.0% 0.0% fover_parse

083f1dd7 2b763188 0.0% 0.0% 0.0% fover_ifc_test

083f56b2 2b762f90 0.0% 0.0% 0.0% fover_health_monitoring_thread

08429600 2b762d98 0.0% 0.0% 0.0% ha_trans_ctl_tx

08429600 2b762ba0 0.0% 0.0% 0.0% ha_trans_data_tx

084213f7 2b7629a8 0.0% 0.0% 0.0% fover_FSM_thread

08dc3b2b 2b7627b0 0.0% 0.0% 0.0% lu_rx

08dc3a5c 2b7625b8 0.0% 0.0% 0.0% lu_dynamic_sync

08b99e4b 2b7623c0 0.0% 0.0% 0.0% SNMP Notify Thread

08cb326d 2b7621c8 0.0% 0.0% 0.0% rtcli async executor process

0852d8e6 2b761fd0 0.0% 0.0% 0.0% IP Thread

0853428e 2b761dd8 0.0% 0.0% 0.0% ARP Thread

08455b80 2b761be0 0.0% 0.0% 0.0% icmp_thread

08c90626 2b7619e8 0.0% 0.0% 0.0% udp_thread

08c6785c 2b7617f0 0.0% 0.0% 0.0% tcp_thread

08c71033 2b7615f8 0.0% 0.0% 0.0% npshim_thread

081e2186 2b761400 0.0% 0.0% 0.0% dns_cache_timer

081dfc3a 2b761208 0.0% 0.0% 0.0% dns_process

08be093b 2b761010 0.0% 0.0% 0.0% EAPoUDP-sock

08215925 2b760e18 0.0% 0.0% 0.0% EAPoUDP

0824b323 2b760c20 0.0% 0.0% 0.0% emweb/https

08241d06 2b760a28 0.0% 0.0% 0.0% Timekeeper

08cb326d 2b760830 0.0% 0.0% 0.0% Unicorn Proxy Thread

08c8f7c8 2b760248 0.0% 0.0% 0.0% snmp

08be093b 2b760050 0.0% 0.0% 0.0% IKE Receiver

08c72144 2b75fe58 0.0% 0.0% 0.0% listen/ssh

081f7fe1 2b75fc60 0.0% 0.0% 0.0% DHCPD Timer

081f9bce 2b75fa68 0.0% 0.0% 0.0% dhcp_daemon

0880bcc3 2b75f870 0.0% 0.0% 0.0% NTP

08cb326d 2b75f678 0.0% 0.0% 0.0% cachefs

08e1f58d 2b75f288 0.0% 0.0% 0.0% vpnfol_thread_msg

08e25bc2 2b75f090 0.0% 0.0% 0.0% vpnfol_thread_timer

08e23e92 2b75ee98 0.0% 0.0% 0.0% vpnfol_thread_sync

08e2574c 2b75eca0 0.0% 0.0% 0.0% vpnfol_thread_unsent

0851f5e8 2b75e8b0 0.0% 0.0% 0.0% Integrity Fw Timer Thread

086a3eec 2b75e6b8 0.0% 0.0% 0.0% netfs_vnode_reclaim

08c27a2b 2b75e0d0 0.0% 0.0% 0.0% ssh/timer

089790f3 2b75bb68 0.0% 0.0% 0.0% vPif_stats_cleaner

0892381a 2b753588 0.1% 0.0% 0.0% ssh

- - 14.9% 15.6% 15.5% DATAPATH-0-461

- - 14.3% 15.3% 15.1% DATAPATH-1-462

- - 14.4% 15.2% 15.1% DATAPATH-2-463

- - 14.5% 15.3% 15.2% DATAPATH-3-464

Show conn count

95027 in use, 937641 most used

show xlate count

3197 in use, 34478 most used

show interface inside

Interface GigabitEthernet3/1 "inside", is up, line protocol is up

Hardware is i82571EB 4CU rev06, BW 1000 Mbps, DLY 10 usec

Auto-Duplex(Full-duplex), Auto-Speed(1000 Mbps)

MAC address 0015.17a0.63b9, MTU 1500

IP address 192.168.16.1, subnet mask 255.255.240.0

243665203 packets input, 42774703631 bytes, 0 no buffer

Received 17406 broadcasts, 0 runts, 0 giants

505 input errors, 0 CRC, 0 frame, 505 overrun, 0 ignored, 0 abort

0 L2 decode drops

238400297 packets output, 43788389697 bytes, 1832 underruns

0 output errors, 0 collisions, 0 interface resets

0 late collisions, 0 deferred

0 input reset drops, 0 output reset drops

input queue (blocks free curr/low): hardware (489/287)

output queue (blocks free curr/low): hardware (493/0)

Traffic Statistics for "inside":

243665209 packets input, 38145576295 bytes

238402138 packets output, 39252496262 bytes

97957 packets dropped

1 minute input rate 162172 pkts/sec, 24694104 bytes/sec

1 minute output rate 157889 pkts/sec, 29197191 bytes/sec

1 minute drop rate, 63 pkts/sec

5 minute input rate 175285 pkts/sec, 26760582 bytes/sec

5 minute output rate 169616 pkts/sec, 30749982 bytes/sec

5 minute drop rate, 65 pkts/sec

show interface outside

Interface GigabitEthernet3/0 "outside", is up, line protocol is up

Hardware is i82571EB 4CU rev06, BW 1000 Mbps, DLY 10 usec

Auto-Duplex(Full-duplex), Auto-Speed(1000 Mbps)

MAC address 0015.17a0.63b8, MTU 1500

IP address xxxxx, subnet mask 255.255.255.0

241555684 packets input, 43843328988 bytes, 0 no buffer

Received 3856 broadcasts, 0 runts, 0 giants

703279 input errors, 0 CRC, 0 frame, 703279 overrun, 0 ignored, 0 abort

0 L2 decode drops

241524745 packets output, 42333605400 bytes, 44 underruns

0 output errors, 0 collisions, 0 interface resets

0 late collisions, 0 deferred

0 input reset drops, 0 output reset drops

input queue (blocks free curr/low): hardware (511/296)

output queue (blocks free curr/low): hardware (503/0)

Traffic Statistics for "outside":

241555678 packets input, 39247672818 bytes

241524797 packets output, 37743585319 bytes

5247768 packets dropped

1 minute input rate 157433 pkts/sec, 29061455 bytes/sec

1 minute output rate 161037 pkts/sec, 24468052 bytes/sec

1 minute drop rate, 675 pkts/sec

5 minute input rate 169368 pkts/sec, 30643618 bytes/sec

5 minute output rate 174233 pkts/sec, 26545770 bytes/sec

5 minute drop rate, 814 pkts/sec

show version

Cisco Adaptive Security Appliance Software Version 8.2(1)

Device Manager Version 6.2(1)

Compiled on Tue 05-May-09 23:45 by builders

System image file is "disk0:/asa821-smp-k8.bin"

Config file at boot was "startup-config"

asa up 17 days 7 hours

Hardware: ASA5580-20, 8192 MB RAM, CPU AMD Opteron 2600 MHz

2 CPUs, 4 cores

Internal ATA Compact Flash, 1024MB

BIOS Flash MX29LV320 @ 0xffc00000, 4096KB

edadios · ‎06-16-2010

There is definite issue being shown with the interface statistics.

Interface ouput queue is showing 0 block for low. Meaning at some time, it run out of allocated memory.

show interface inside

output queue (blocks free curr/low): hardware (493/0)

There is also a lot of overruns on the outside interface

show interface outside

703279 input errors, 0 CRC, 0 frame, 703279 overrun, 0 ignored, 0 abort

output queue (blocks free curr/low): hardware (503/0)

It will need to be investigated further. Possibly best when the blocks free current is really showing low, or when the overrun are incrementing.

Need to check/classify the traffic, and see if you can find some sort of pattern in there for what may be adding to the slowness.

For your reference, here is the datasheet that includes the 5580-20 information :

http://www.cisco.com/en/US/customer/prod/collateral/vpndevc/ps6032/ps6094/ps6120/product_data_sheet0900aecd802930c5.html

If you need further assistance, I recommend logging an SR with TAC.

Regards,

inteltechs · ‎06-16-2010

we are getting 1.5+ billion searches inbound and 2+ billion outbound requests daily. These are search traffic so it happens very fast. By the way, how can I increase the allocated memory? This box still has plenty of memory available I believe.

Free memory: 3042781768 bytes (71%)

Used memory: 1252185527 bytes (29%)

show interface outside

Interface GigabitEthernet3/0 "outside", is up, line protocol is up

Hardware is i82571EB 4CU rev06, BW 1000 Mbps, DLY 10 usec

Auto-Duplex(Full-duplex), Auto-Speed(1000 Mbps)

MAC address 0015.17a0.63b8, MTU 1500

IP address xxxxxxxx, subnet mask 255.255.255.0

1908656309 packets input, 374595776114 bytes, 0 no buffer

Received 27679 broadcasts, 0 runts, 0 giants

5129638 input errors, 0 CRC, 0 frame, 5129638 overrun, 0 ignored, 0 abort

0 L2 decode drops

1947847973 packets output, 335474211179 bytes, 1001 underruns

0 output errors, 0 collisions, 0 interface resets

0 late collisions, 0 deferred

0 input reset drops, 0 output reset drops

input queue (blocks free curr/low): hardware (460/296)

output queue (blocks free curr/low): hardware (438/0)

Traffic Statistics for "outside":

1908656291 packets input, 338189002502 bytes

1947848980 packets output, 298213928906 bytes

15253521 packets dropped

1 minute input rate 157529 pkts/sec, 27078482 bytes/sec

1 minute output rate 161482 pkts/sec, 24349800 bytes/sec

1 minute drop rate, 1258 pkts/sec

5 minute input rate 158636 pkts/sec, 27594895 bytes/sec

5 minute output rate 162783 pkts/sec, 24556694 bytes/sec

5 minute drop rate, 1384 pkts/sec

bigdaddy# show interface ins

Interface GigabitEthernet3/1 "inside", is up, line protocol is up

Hardware is i82571EB 4CU rev06, BW 1000 Mbps, DLY 10 usec

Auto-Duplex(Full-duplex), Auto-Speed(1000 Mbps)

MAC address 0015.17a0.63b9, MTU 1500

IP address 192.168.16.1, subnet mask 255.255.240.0

1961019889 packets input, 338346563327 bytes, 0 no buffer

Received 125888 broadcasts, 0 runts, 0 giants

3707 input errors, 0 CRC, 0 frame, 3707 overrun, 0 ignored, 0 abort

0 L2 decode drops

1906263525 packets output, 375441157497 bytes, 5281 underruns

0 output errors, 0 collisions, 0 interface resets

0 late collisions, 0 deferred

0 input reset drops, 0 output reset drops

input queue (blocks free curr/low): hardware (497/287)

output queue (blocks free curr/low): hardware (451/0)

Traffic Statistics for "inside":

1961019895 packets input, 300846990614 bytes

1906268813 packets output, 339101732071 bytes

941279 packets dropped

1 minute input rate 162410 pkts/sec, 24532489 bytes/sec

1 minute output rate 157133 pkts/sec, 27132857 bytes/sec

1 minute drop rate, 119 pkts/sec

5 minute input rate 163716 pkts/sec, 24749199 bytes/sec

5 minute output rate 158122 pkts/sec, 27621085 bytes/sec

5 minute drop rate, 152 pkts/sec

edadios · ‎06-16-2010

There is no configuration to do, to set the block on interfaces.

You will need to do some sort of baseline, of possible times when all is good through the firewall, check for firewall statistices then, and do continung trend of traffic through your firewall.

If you are not able to characterize the traffic that may be contributing to this, or maybe work out from logs about possible malicious traffic that may be contributing to this, then I suggest you open an SR with TAC to troubleshoot further with you.