09-23-2013 08:53 PM - edited 03-11-2019 07:42 PM
Hello,
Yesterday the ASA 5585-X S40 going up to 99% cpu, I was checked almost ACL, service policy, disabled threat-detection but can't decrease cpu load. I tried some discussion, document but no luck.
Result of the command: "show cpu usage"
CPU utilization for 5 seconds = 99%; 1 minute: 99%; 5 minutes: 99%
Result of the command: "show processes cpu-usage"
PC Thread 5Sec 1Min 5Min Process
0x0000000000c30a1a 0x00007ffdf6cd5fd0 0.0% 0.0% 0.0% WebVPN KCD Process
0x000000000180e214 0x00007ffdf6cc1260 0.0% 0.0% 0.0% vpnlb_timer_thread
0x000000000180ea2a 0x00007ffdf6cc5e00 0.0% 0.0% 0.0% vpnlb_thread
0x000000000160c61f 0x00007ffdf6cd4ea0 0.0% 0.0% 0.0% UserFromCert Thread
0x000000000116007a 0x00007ffdf6cb88e0 0.0% 0.0% 0.0% update_cpu_usage
0x000000000160c61f 0x00007ffdf6cb1770 0.0% 0.0% 0.0% Unicorn Proxy Thread
0x0000000001d44c62 0x00007ffdf6c95c00 0.0% 0.0% 0.0% Unicorn Admin Handler
0x00000000015acf40 0x00007ffdf6cd3690 0.1% 0.1% 0.1% Unicorn Admin Handler
0x0000000001d57530 0x00007ffdf6c9c320 0.1% 0.1% 0.1% Unicorn Admin Handler
0x00000000015acf40 0x00007ffdf6ccc890 0.0% 0.0% 0.0% Unicorn Admin Handler
0x00000000015acf40 0x00007ffdf6c92870 0.1% 0.1% 0.1% Unicorn Admin Handler
0x0000000001d57530 0x00007ffdf6ca5380 0.1% 0.1% 0.1% Unicorn Admin Handler
0x00000000015acf40 0x00007ffdf6ccc1b0 0.1% 0.1% 0.1% Unicorn Admin Handler
0x00000000010ca4e5 0x00007ffdf6c9bc40 0.0% 0.0% 0.0% Unicorn Admin Handler
0x00000000015e7fa2 0x00007ffdf6cc2390 0.0% 0.0% 0.0% udp_timer
0x00000000015e8d5c 0x00007ffdf6cb2c10 0.0% 0.0% 0.0% udp_thread
0x00000000011443af 0x00007ffdf6cbd110 0.0% 0.0% 0.0% uauth_urlb clean
0x0000000001523c1e 0x00007ffdf6cc9190 0.0% 0.0% 0.0% Uauth_Proxy
0x000000000159bed5 0x00007ffdf6cc9500 0.0% 0.0% 0.0% uauth
0x00000000006529a1 0x00007ffdf6cc04a0 0.0% 0.0% 0.0% tmatch compile thread
0x0000000000782fd5 0x00007ffdf6cc4280 0.0% 0.0% 0.0% TLS Proxy Inspector
0x0000000000701af5 0x00007ffdf6cb1ae0 0.0% 0.0% 0.0% Timekeeper
0x000000000158cffd 0x00007ffdf6cc7980 0.0% 0.0% 0.0% Thread Logger
0x00000000015a4cad 0x00007ffdf6cb28a0 0.0% 0.0% 0.0% tcp_thread
0x00000000015a9895 0x00007ffdf6cc2700 0.0% 0.0% 0.0% tcp_slow
0x00000000015a17bb 0x00007ffdf6cc2a70 0.0% 0.0% 0.0% tcp_fast
0x0000000001597a27 0x00007ffdf6cc7610 0.0% 0.0% 0.0% syslogd
0x0000000001591fbd 0x00007ffdf6cc7cf0 0.0% 0.0% 0.0% Syslog Retry Thread
0x00000000005c86e5 0x00007ffdf6cd21f0 0.0% 0.0% 0.0% SXP CORE
0x000000000156c25d 0x00007ffdf6cc8740 0.0% 0.0% 0.0% SSL
0x00000000015558ad 0x00007ffdf6c94760 0.0% 0.0% 0.0% ssh/timer
0x000000000078d6fa 0x00007ffdf6cba460 0.0% 0.0% 0.0% SRTP Processing
0x000000000148a68a 0x00007ffdf6cbec90 0.0% 0.0% 0.0% snmpfo_timer_thread
0x0000000001489e9a 0x00007ffdf6cb3d40 0.0% 0.0% 0.0% SNMP Notify Thread
0x0000000001599b8a 0x00007ffdf6cc83d0 0.0% 0.0% 0.0% SMTP
0x000000000143a6b3 0x00007ffdf6cc9be0 0.0% 0.0% 0.0% Session Manager
0x000000000184301b 0x00007ffdf6cc4cd0 0.0% 0.0% 0.0% scansafe_poll
0x000000000160c61f 0x00007ffdf6cb40b0 0.0% 0.0% 0.0% rtcli async executor process
0x000000000043192d 0x00007ffdf6cd5580 0.0% 0.0% 0.0% Reload Control Thread
0x00000000005c091c 0x00007ffdf6cd1e80 0.0% 0.0% 0.0% RBM CORE
0x000000000153c6b5 0x00007ffdf6cbb220 0.0% 0.0% 0.0% RADIUS Proxy Time Keeper
0x00000000014fb13e 0x00007ffdf6cbb590 0.0% 0.0% 0.0% RADIUS Proxy Listener
0x000000000153d0cd 0x00007ffdf6cbb900 0.0% 0.0% 0.0% RADIUS Proxy Event Daemon
0x00000000013b9f95 0x00007ffdf6cc9f50 0.0% 0.0% 0.0% Quack process
0x0000000001365acf 0x00007ffdf6c97780 0.0% 0.0% 0.0% qos_metric_daemon
0x0000000001391d2d 0x00007ffdf6ccb3f0 0.0% 0.0% 0.0% QoS Support Module
0x000000000179afda 0x00007ffdf6cc15d0 0.0% 0.0% 0.0% ppp_timer_thread
0x000000000111518a 0x00007ffdf6cbcda0 0.0% 0.0% 0.0% pm_timer_thread
0x000000000112dfbf 0x00007ffdf6ccbad0 0.0% 0.0% 0.0% PIX Garbage Collector
0x000000000183bd9a 0x00007ffdf6cc5040 0.0% 0.0% 0.0% pci_nt_bridge
0x0000000000f4559b 0x00007ffdf6ca8710 0.0% 0.0% 0.0% NTP
0x00000000015bb10d 0x00007ffdf6cb8570 0.1% 0.1% 0.1% npshim_thread
0x000000000116011a 0x00007ffdf6cb7e90 0.0% 0.0% 0.0% NIC status poll
0x0000000000d3e0c3 0x00007ffdf6caff60 0.0% 0.0% 0.0% netfs_vnode_reclaim
0x000000000160c61f 0x00007ffdf6ccdd30 0.0% 0.0% 0.0% netfs_thread_init
0x0000000000d3dd71 0x00007ffdf6cc3ba0 0.0% 0.0% 0.0% netfs_mount_handler
0x0000000001780313 0x00007ffdf6cb4790 0.0% 0.0% 0.0% lu_rx
0x0000000001780352 0x00007ffdf6cb4420 0.0% 0.0% 0.0% lu_dynamic_sync
0x000000000178041f 0x00007ffdf6cb8c50 0.0% 0.0% 0.0% lu_ctl
0x0000000001593a32 0x00007ffdf6cc8060 0.1% 0.1% 0.1% Logger
0x00000000015b1960 0x00007ffdf6cb0d20 0.0% 0.0% 0.0% listen/telnet
0x00000000015b1960 0x00007ffdf6cb09b0 0.0% 0.0% 0.0% listen/ssh
0x0000000000c3169a 0x00007ffdf6cc3150 0.0% 0.0% 0.0% Lic TMR
0x0000000000c3145b 0x00007ffdf6cc2de0 0.0% 0.0% 0.0% Lic HA Cluster
0x0000000000c69f64 0x00007ffdf6cb39d0 0.0% 0.0% 0.0% LACP Protocol
0x00000000017af8a5 0x00007ffdf6cc1940 0.0% 0.0% 0.0% L2TP mgmt daemon
0x00000000017ad0c5 0x00007ffdf6cc1cb0 0.0% 0.0% 0.0% L2TP data daemon
0x000000000052ec2c 0x00007ffdf6cc0ef0 0.0% 0.0% 0.0% IPsec message handler
0x0000000000b0e51d 0x00007ffdf6cb3660 0.0% 0.0% 0.0% IP Thread
0x0000000000f6fc4b 0x00007ffdf6cbfa50 0.0% 0.0% 0.0% IP Background
0x000000000110ac15 0x00007ffdf6ccb760 0.0% 0.0% 0.0% IP Address Assign
0x000000000094ba6a 0x00007ffdf6cd6340 0.0% 0.0% 0.0% intel_82574_esd_detection_thread
0x0000000000aff815 0x00007ffdf6cb02d0 0.0% 0.0% 0.0% Integrity Fw Timer Thread
0x0000000000b02e0a 0x00007ffdf6cbaeb0 0.0% 0.0% 0.0% Integrity FW Task
0x0000000000a8a759 0x00007ffdf6cbbc70 0.0% 0.0% 0.0% IKEv2 DPD Client Process
0x0000000000a8b137 0x00007ffdf6cbbfe0 0.0% 0.0% 0.0% IKEv2 Daemon
0x00000000009ed3e5 0x00007ffdf6cbc6c0 0.0% 0.0% 0.0% IKE Timekeeper
0x00000000009dd31a 0x00007ffdf6cbc350 0.0% 0.0% 0.0% IKE Daemon
0x00000000009b7d45 0x00007ffdf6cbca30 0.0% 0.0% 0.0% IKE Common thread
0x00000000010a9ead 0x00007ffdf6cccf70 0.0% 0.0% 0.0% idfw_service
0x000000000109d88d 0x00007ffdf6ccd2e0 0.0% 0.0% 0.0% idfw_proc
0x00000000010b58cd 0x00007ffdf6cccc00 0.0% 0.0% 0.0% idfw_adagent
0x00000000009575fd 0x00007ffdf6cb2f80 0.0% 0.0% 0.0% icmp_thread
0x00000000013ec4f5 0x00007ffdf6cc0130 0.0% 0.0% 0.0% ICMP event handler
0x000000000092b1bc 0x00007ffdf6cb4e70 0.0% 0.0% 0.0% ha_trans_data_tx
0x000000000092b1bc 0x00007ffdf6cb51e0 0.0% 0.0% 0.0% ha_trans_ctl_tx
0x000000000090cc48 0x00007ffdf6cb69f0 0.0% 0.0% 0.0% fover_tx_2
0x000000000090cc48 0x00007ffdf6cb6d60 0.0% 0.0% 0.0% fover_tx
0x000000000091117e 0x00007ffdf6cb8fc0 0.0% 0.0% 0.0% fover_thread
0x000000000090b1e2 0x00007ffdf6cb70d0 0.0% 0.0% 0.0% fover_rx
0x0000000000915aca 0x00007ffdf6cb6310 0.0% 0.0% 0.0% fover_rep
0x000000000090b0cf 0x00007ffdf6cb5fa0 0.0% 0.0% 0.0% fover_parse
0x00000000008fddb2 0x00007ffdf6cb6680 0.0% 0.0% 0.0% fover_ip
0x00000000008e92f5 0x00007ffdf6cb58c0 0.0% 0.0% 0.0% fover_ifc_test
0x00000000008ed006 0x00007ffdf6cb5550 0.0% 0.0% 0.0% fover_health_monitoring_thread
0x0000000000921a0d 0x00007ffdf6cb4b00 0.0% 0.0% 0.0% fover_FSM_thread
0x00000000008ece25 0x00007ffdf6cb5c30 0.0% 0.0% 0.0% fover_fail_check
0x000000000071f15a 0x00007ffdf6cd6a20 0.0% 0.0% 0.0% Environment Monitor Process
0x000000000070c500 0x00007ffdf6cb1e50 0.0% 0.0% 0.0% emweb/https
0x000000000145739a 0x00007ffdf6cc3f10 0.0% 0.0% 0.0% emweb/cifs_timer
0x00000000014fb13e 0x00007ffdf6cb2530 0.0% 0.0% 0.0% EAPoUDP-sock
0x00000000006d35f5 0x00007ffdf6cb21c0 0.0% 0.0% 0.0% EAPoUDP
0x0000000001766675 0x00007ffdf6cbfdc0 0.0% 0.0% 0.0% Dynamic Filter VC Housekeeper
0x000000000068814a 0x00007ffdf6cd09e0 0.0% 0.0% 0.0% dbgtrace
0x00000000005b91f5 0x00007ffdf6cd17a0 0.0% 0.0% 0.0% cts_timer_task
0x00000000005b7c12 0x00007ffdf6cd1b10 0.0% 0.0% 0.0% cts_task
0x000000000054b982 0x00007ffdf6cc0b80 0.0% 0.0% 0.0% CTM message handler
0x0000000000544b08 0x00007ffdf6cd2560 0.0% 0.0% 0.0% CTM Daemon
0x00000000004fc3a5 0x00007ffdf6cc2020 0.0% 0.0% 0.0% CTCP Timer process
0x000000000129e025 0x00007ffdf6cbf000 0.0% 0.0% 0.0% Crypto PKI RECV
0x000000000129f74e 0x00007ffdf6cbf6e0 0.0% 0.0% 0.0% Crypto CA
0x000000000078dc8f 0x00007ffdf6cb96a0 0.0% 0.0% 0.0% CP Threat-Detection Processing
0x000000000078d9d8 0x00007ffdf6cbab40 0.1% 0.1% 0.1% CP Processing
0x000000000078d818 0x00007ffdf6cba7d0 0.0% 0.0% 0.0% CP Midpath Processing
0x000000000078d60f 0x00007ffdf6cba0f0 0.0% 0.0% 0.0% CP HA Processing
0x000000000078dac7 0x00007ffdf6cb9a10 0.0% 0.0% 0.0% CP DP CXSC Event Processing
0x000000000078d522 0x00007ffdf6cc0810 0.0% 0.0% 0.0% CP Crypto Result Processing
0x000000000078dbaa 0x00007ffdf6cb9d80 0.0% 0.0% 0.0% CP ARP Processing
0x0000000000489b75 0x00007ffdf6cd40e0 0.0% 0.0% 0.0% CMGR Timer Process
0x00000000004875da 0x00007ffdf6cd4450 0.0% 0.0% 0.0% CMGR Server Process
0x000000000048649b 0x00007ffdf6cd3d70 0.0% 0.0% 0.0% CMGR Proxy Process
0x00000000005f3de5 0x00007ffdf6ccf540 0.0% 0.0% 0.0% cluster interface health monitor
0x00000000011bf345 0x00007ffdf6ccb080 0.0% 0.0% 0.0% Client Update Task
0x00000000006a7987 0x00007ffdf6cb9330 0.0% 0.0% 0.0% ci/console
0x0000000001c7a6c5 0x00007ffdf6ccbe40 0.0% 0.0% 0.0% Chunk Manager
0x0000000001d880e5 0x00007ffdf6ccad10 0.0% 0.0% 0.0% Checkheaps
0x00000000012e0588 0x00007ffdf6cbf370 0.0% 0.0% 0.0% CERT API
0x000000000041858a 0x00007ffdf6cd7b50 0.0% 0.0% 0.0% block_diag
0x0000000001d57578 0x00007ffdf6cbded0 0.0% 0.0% 0.0% bcmXGS3AsyncTX
0x0000000001d57578 0x00007ffdf6cbe240 0.0% 0.0% 0.0% bcmTX
0x00000000021a9255 0x00007ffdf6cc45f0 0.0% 0.0% 0.0% bcmPOLL
0x0000000001d575be 0x00007ffdf6cbdb60 0.1% 0.1% 0.1% bcmLINK.0
0x0000000001d57578 0x00007ffdf6cd7100 0.0% 0.0% 0.0% bcmDPC
0x00000000021a9202 0x00007ffdf6cbe5b0 0.0% 0.0% 0.0% bcmCNTR.0
0x0000000001d1f69a 0x00007ffdf6cd7470 0.0% 0.0% 0.0% bcm_esw_surprise_reset.0
0x0000000001d28288 0x00007ffdf6cbd7f0 0.0% 0.0% 0.0% bcm_esw_poll
0x0000000001d25d58 0x00007ffdf6cbd480 0.0% 0.0% 0.0% bcm_esw_ioctl_task_handler
0x0000000000b14899 0x00007ffdf6cc3830 0.0% 0.0% 0.0% arp_timer
0x0000000000b1ef95 0x00007ffdf6cc34c0 0.0% 0.0% 0.0% arp_forward_thread
0x0000000000b1616d 0x00007ffdf6cb32f0 0.0% 0.0% 0.0% ARP Thread
0x000000000160c61f 0x00007ffdf6cd4b30 0.0% 0.0% 0.0% aaa_shim_thread
0x0000000000446088 0x00007ffdf6cd5210 0.0% 0.0% 0.0% aaa
- - 6.1% 6.2% 6.2% DATAPATH-0-2013
- - 6.2% 6.2% 6.2% DATAPATH-1-2014
- - 6.2% 6.2% 6.2% DATAPATH-2-2015
- - 6.2% 6.2% 6.2% DATAPATH-3-2016
- - 6.2% 6.2% 6.2% DATAPATH-4-2017
- - 6.2% 6.2% 6.2% DATAPATH-5-2018
- - 6.2% 6.2% 6.2% DATAPATH-6-2019
- - 6.2% 6.2% 6.2% DATAPATH-7-2020
- - 6.2% 6.2% 6.2% DATAPATH-8-2021
- - 6.2% 6.2% 6.2% DATAPATH-9-2022
- - 6.2% 6.2% 6.2% DATAPATH-10-2023
- - 6.2% 6.2% 6.2% DATAPATH-11-2024
- - 6.1% 6.1% 6.1% DATAPATH-12-2025
- - 6.2% 6.2% 6.2% DATAPATH-13-2026
- - 6.2% 6.2% 6.2% DATAPATH-14-2027
- - 6.2% 6.2% 6.2% DATAPATH-15-2028
Result of the command: "Show conn count"
35053 in use, 1130273 most used
Result of the command: "show version"
Cisco Adaptive Security Appliance Software Version 9.1(3)
Device Manager Version 7.1(4)
Compiled on Mon 16-Sep-13 16:07 PDT by builders
System image file is "disk0:/asa913-smp-k8.bin"
Config file at boot was "startup-config"
FIREWALL up 4 days 23 hours
Hardware: ASA5585-SSP-40
Result of the command: "show interface"
Interface Port-channel1 "outside", is up, line protocol is up
Hardware is EtherChannel/LACP, BW 2000 Mbps, DLY 10 usec
Auto-Duplex(Full-duplex), Auto-Speed(1000 Mbps)
Input flow control is unsupported, output flow control is off
MAC address ccef.482d.958e, MTU 1500
IP address xxx.xxx.xxx.xxx, subnet mask 255.255.255.0
Traffic Statistics for "outside":
47856250231 packets input, 3385651497302 bytes
43971669158 packets output, 24327589398287 bytes
10770939705 packets dropped
1 minute input rate 71403 pkts/sec, 6373285 bytes/sec
1 minute output rate 85420 pkts/sec, 55197049 bytes/sec
1 minute drop rate, 294 pkts/sec
5 minute input rate 70846 pkts/sec, 6859039 bytes/sec
5 minute output rate 83780 pkts/sec, 54740252 bytes/sec
5 minute drop rate, 309 pkts/sec
Members in this channel:
Active: Gi0/0 Gi0/1
Interface Port-channel2 "inside", is up, line protocol is up
Hardware is EtherChannel/LACP, BW 2000 Mbps, DLY 10 usec
Auto-Duplex(Full-duplex), Auto-Speed(1000 Mbps)
Input flow control is unsupported, output flow control is off
MAC address ccef.482d.9592, MTU 1500
IP address xxx.xxx.xxx.xxx, subnet mask 255.255.255.0
Traffic Statistics for "inside":
44006373346 packets input, 24068117518551 bytes
36809486649 packets output, 2853674226726 bytes
781620377 packets dropped
1 minute input rate 84992 pkts/sec, 55168999 bytes/sec
1 minute output rate 70300 pkts/sec, 6115406 bytes/sec
1 minute drop rate, 1371 pkts/sec
5 minute input rate 83793 pkts/sec, 54533949 bytes/sec
5 minute output rate 69850 pkts/sec, 6802253 bytes/sec
5 minute drop rate, 1457 pkts/sec
Members in this channel:
Active: Gi0/4 Gi0/5
09-24-2013 02:02 AM
Result of the command: "show proc cpu-usage non-zero sorted"
PC Thread 5Sec 1Min 5Min Process
- - 6.2% 6.1% 6.1% DATAPATH-8-2021
- - 6.2% 6.1% 6.1% DATAPATH-0-2013
- - 6.2% 6.1% 6.1% DATAPATH-12-2025
- - 6.1% 6.1% 6.1% DATAPATH-7-2020
- - 6.1% 6.1% 6.1% DATAPATH-9-2022
- - 6.1% 6.1% 6.1% DATAPATH-10-2023
- - 6.1% 6.1% 6.1% DATAPATH-11-2024
- - 6.1% 6.1% 6.1% DATAPATH-1-2014
- - 6.1% 6.1% 6.1% DATAPATH-2-2015
- - 6.1% 6.1% 6.1% DATAPATH-5-2018
- - 6.1% 6.1% 6.1% DATAPATH-15-2028
- - 6.1% 6.1% 6.1% DATAPATH-4-2017
- - 6.1% 6.1% 6.1% DATAPATH-6-2019
- - 6.1% 6.1% 6.1% DATAPATH-13-2026
- - 6.1% 6.1% 6.1% DATAPATH-14-2027
- - 6.1% 6.1% 6.1% DATAPATH-3-2016
0x0000000001d57530 0x00007ffdf6c9e580 0.1% 0.1% 0.1% Unicorn Admin Handler
0x0000000001593a32 0x00007ffdf6cc8060 0.1% 0.1% 0.1% Logger
0x00000000015bb10d 0x00007ffdf6cb8570 0.1% 0.1% 0.1% npshim_thread
0x00000000015acf40 0x00007ffdf6cd3690 0.1% 0.1% 0.1% Unicorn Admin Handler
0x00000000021a9202 0x00007ffdf6cbdb60 0.1% 0.1% 0.1% bcmLINK.0
0x00000000015acf40 0x00007ffdf6c9e210 0.1% 0.1% 0.1% Unicorn Admin Handler
0x000000000078d9d8 0x00007ffdf6cbab40 0.1% 0.1% 0.1% CP Processing
0x00000000010ef725 0x00007ffdf6c9c320 0.1% 0.1% 0.1% Unicorn Admin Handler
0x00000000015acf40 0x00007ffdf6c97af0 0.1% 0.1% 0.1% Unicorn Admin Handler
0x000000000078d818 0x00007ffdf6cba7d0 0.1% 0.1% 0.1% CP Midpath Processing
0x0000000001d57530 0x00007ffdf6c92500 0.0% 0.1% 0.1% Unicorn Admin Handler
Who can tell me what is DATAPATH process and how to trace it ?
09-24-2013 06:08 AM
Hi,
These kind of problems are usually best handled through the Cisco TAC (for which you need the service contract)
- Jouni
10-03-2013 03:13 AM
Hi, Have you had any updates from TAC around this issue? i have a similar issue on 5585-X S60 clustering pair running 9.1(2)
Thanks
David
Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: