Cisco Support Community
cancel
Showing results for 
Search instead for 
Did you mean: 
New Member

Dispatch Unit - High CPU

Hi All,

I'm trying to do some research on the Dispatch Unit process.  It seems High CPU and this process go hand in hand.  I haven't figured out an effective way of determining what underlying issue is the actual source.  Can someone point me in the right direction to try an understand what the Dispatch Unit process is doing?  I have an ASA 5550.  I have seen the cpu hover around 85% +- 5% for sustained long periods, 30 - 60 min +.  I have always been under the impression that around 80% cpu and you're probably dropping packets (that could be an out-dated belief).

Any help to understand this is much appreciated.

-E

Everyone's tags (2)
1 ACCEPTED SOLUTION

Accepted Solutions
Cisco Employee

Re: Dispatch Unit - High CPU

Hello Edward,

That is indeed very true, but that only aleaveates the CPU processing but it does not get rid of the issue, based on the ICMP messages and netbios, there could be loops and packets bouncing, hence the high amount of inspection hits. Adding to the packet processing on the interface itself, the inspection will add another CPU intensing consuming.

As you rightly pointed, the main thing on these cases is to first identify the traffic and then, check if it is normal or not.

Mike

Mike
62 REPLIES
Cisco Employee

Dispatch Unit - High CPU

Edward,

Dispatch unit is the central packet processing process, it does interface polling and MULTIPLE check on ASP.

Typically when your high cpu is in dispatch, you need to look into traffic pattern, enabling unicast RPF on all interfaces is a good start.

"show int" and "show traff" , "show perfmon" to understand what traffic and on which interface is causing this problem.

M.

New Member

Dispatch Unit - High CPU

Marcin,

Thank you for the response.  I was curious if the ASP "process" was happening within the dispatch unit process.  This helps.  I did do some more research on unicast RPF.  This will take some more research.  I've found some good documentation and thank you for that direction.

My interface statistics were clean.

CPU utilization for 5 seconds = 80%; 1 minute: 79%; 5 minutes: 74%

show conn count

76078 in use, 139617 most used

show proc cpu-usage sorted non-zero

PC         Thread       5Sec     1Min     5Min   Process

0x081d8531   0x1bdc1528    77.2%    78.0%    73.9%   Dispatch Unit

0x08ead27c   0x1bdb9d50     0.8%     0.7%     0.7%   Logger

0x08ef7d09   0x1bdae270     0.2%     0.2%     0.2%   snmp

0x08e8f4ec   0x1bdc0b00     0.2%     0.2%     0.2%   ssm4ge_cfg_poll_thread

0x08e6c2f5   0x1bd9de70     0.1%     0.0%     0.0%   ssh

Show traffic, the outside int was the busiest:

Outside:

        received (in 15571.990 secs):

                77762147 packets        12436865813 bytes

                4166 pkts/sec   798117 bytes/sec

        transmitted (in 15571.990 secs):

                58494240 packets        57798648078 bytes

                3204 pkts/sec   3711154 bytes/sec

      1 minute input rate 2712 pkts/sec,  399665 bytes/sec

      1 minute output rate 3130 pkts/sec,  3244847 bytes/sec

      1 minute drop rate, 40 pkts/sec

      5 minute input rate 4025 pkts/sec,  1922721 bytes/sec

      5 minute output rate 4075 pkts/sec,  3665981 bytes/sec

      5 minute drop rate, 45 pkts/sec

PERFMON STATS:                     Current      Average

Xlates                                2/s          7/s

Connections                         744/s          6/s

TCP Conns                           470/s          9/s

UDP Conns                           259/s          6/s

URL Access                            0/s          0/s

URL Server Req                        0/s          0/s

TCP Fixup                             0/s          0/s

TCP Intercept Established Conns       0/s          0/s

TCP Intercept Attempts                0/s          0/s

TCP Embryonic Conns Timeout           1/s          4/s

HTTP Fixup                            0/s          0/s

FTP Fixup                             0/s          0/s

AAA Authen                            0/s          0/s

AAA Author                            0/s          0/s

AAA Account                           0/s          0/s

New Member

Re: Dispatch Unit - High CPU

Hi,

I agree with Marcin. Share those outputs here to help you further.

Anton

Sent from Cisco Technical Support iPad App

New Member

Dispatch Unit - High CPU

Edward,

Plese share the output of "sh service-policy"

Regards,

Anton

New Member

Dispatch Unit - High CPU

I didn't clear these statistics.  These are probably since the last power on.

Global policy:

  Service-policy: global_policy

    Class-map: inspection_default

      Inspect: ftp, packet 263278, drop 0, reset-drop 0

      Inspect: h323 h225 _default_h323_map, packet 0, drop 0, reset-drop 0

               tcp-proxy: bytes in buffer 0, bytes dropped 0

      Inspect: h323 ras _default_h323_map, packet 0, drop 0, reset-drop 0

      Inspect: rsh, packet 0, drop 0, reset-drop 0

      Inspect: rtsp, packet 0, drop 0, reset-drop 0

               tcp-proxy: bytes in buffer 0, bytes dropped 0

      Inspect: sqlnet, packet 0, drop 0, reset-drop 0

      Inspect: skinny , packet 0, drop 0, reset-drop 0

               tcp-proxy: bytes in buffer 0, bytes dropped 0

      Inspect: sunrpc, packet 12680779, drop 0, reset-drop 0

               tcp-proxy: bytes in buffer 0, bytes dropped 0

      Inspect: xdmcp, packet 0, drop 0, reset-drop 0

      Inspect: sip , packet 0, drop 0, reset-drop 0

               tcp-proxy: bytes in buffer 0, bytes dropped 0

      Inspect: netbios, packet 637867192, drop 0, reset-drop 0

      Inspect: tftp, packet 0, drop 0, reset-drop 0

      Inspect: icmp, packet 10303228, drop 0, reset-drop 0

      Inspect: icmp error, packet 948054, drop 18, reset-drop 0

      Inspect: ip-options _default_ip_options_map, packet 0, drop 0, reset-drop 0

    Class-map: global-class

New Member

Dispatch Unit - High CPU

Edward,

I believe you are using inspection_default. If yes, issue below command. This will not tamper your production. Lets observe the CPU for another half an hour

policy-map global_policy

        class inspection_default

          no inspect icmp

          no inspect icmp error

Regards,

Anton

New Member

Dispatch Unit - High CPU

Friend,

Is the CPU utilization back to normal?

Regards,

Anton

New Member

Dispatch Unit - High CPU

Hey Integreon,

I have same issue here's my default global policy,

Inspect: icmp error, packet 12451113, drop 40170, reset-drop 0

Inspect: icmp, packet 317250117, drop 20057, reset-drop 0

Is droping packet making cpu utilization higher?

Regards,

Arshad Ahmed

New Member

Re: Dispatch Unit - High CPU

Hi Arshad,

Not the drops, but the inspection. What is your ASA software version?

Sent from Cisco Technical Support iPad App

New Member

Dispatch Unit - High CPU

Hi integreon,

Heres my software version.

Cisco Adaptive Security Appliance Software Version 8.2(1)11

Device Manager Version 6.2(3)

Regards,

Arshad Ahmed

New Member

Dispatch Unit - High CPU

Anton,

Thank you for the suggestion.  It will take some time before I can implement the changes.  I will post back the out come once I have made the change.

New Member

Dispatch Unit - High CPU

I removed both inspect icmp and icmp error:

Global policy:

  Service-policy: global_policy

    Class-map: inspection_default

      Inspect: ftp, packet 366, drop 0, reset-drop 0

      Inspect: h323 h225 _default_h323_map, packet 0, drop 0, reset-drop 0

               tcp-proxy: bytes in buffer 0, bytes dropped 0

      Inspect: h323 ras _default_h323_map, packet 0, drop 0, reset-drop 0

      Inspect: rsh, packet 0, drop 0, reset-drop 0

      Inspect: rtsp, packet 0, drop 0, reset-drop 0

               tcp-proxy: bytes in buffer 0, bytes dropped 0

      Inspect: sqlnet, packet 0, drop 0, reset-drop 0

      Inspect: skinny , packet 0, drop 0, reset-drop 0

               tcp-proxy: bytes in buffer 0, bytes dropped 0

      Inspect: sunrpc, packet 41429, drop 0, reset-drop 0

               tcp-proxy: bytes in buffer 0, bytes dropped 0

      Inspect: xdmcp, packet 0, drop 0, reset-drop 0

      Inspect: sip , packet 0, drop 0, reset-drop 0

               tcp-proxy: bytes in buffer 0, bytes dropped 0

      Inspect: netbios, packet 106690219, drop 0, reset-drop 0

      Inspect: tftp, packet 0, drop 0, reset-drop 0

      Inspect: ip-options _default_ip_options_map, packet 0, drop 0, reset-drop 0

    Class-map: global-class

The cpu is still in the 80% range.  What struck me as odd was the number of inspected netbios packets.

New Member

Dispatch Unit - High CPU

OK. Try to remove netbios too. Also what is your ASA version? This might be because of a cosmetic bug too.

New Member

Dispatch Unit - High CPU

8.4(2)

I tried removing and adding the different inspect commands to see if they were in direct relation to cpu usage but it didn't follow.  Unless there is a bug, i think i need to look more into the traffic hitting the asa that is being processed and tune the configs from there.

Dispatch Unit - High CPU

Hello Edward,

First of all ICMP inspect is not by default on the ASA.

Now I would like to see a: show interface, so we can see if there are overruns or underruns.

I think you  haved   oversubscribed your ASA, we will need to check that by determining the througthput of your ASA, so please provide the sh interface.

Regards,

Do rate helpful posts

Julio

Julio Carvajal
Senior Network Security and Core Specialist
CCIE #42930, 2xCCNP, JNCIP-SEC
New Member

Dispatch Unit - High CPU

Interface GigabitEthernet0/0 "Outside", is up, line protocol is up

  Hardware is i82546GB rev03, BW 1000 Mbps, DLY 10 usec

        Auto-Duplex(Full-duplex), Auto-Speed(1000 Mbps)

        Input flow control is unsupported, output flow control is off

        MAC address 503d.e51e.ca06, MTU 1500

        IP address X.X.X.X, subnet mask 255.255.255.128

        533966821 packets input, 97651127709 bytes, 0 no buffer

        Received 281755 broadcasts, 0 runts, 0 giants

        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

        0 pause input, 0 resume input

        0 L2 decode drops

        531835755 packets output, 523869066435 bytes, 254 underruns

        0 pause output, 0 resume output

        0 output errors, 0 collisions, 0 interface resets

        0 late collisions, 0 deferred

        0 input reset drops, 0 output reset drops, 0 tx hangs

        input queue (blocks free curr/low): hardware (255/230)

        output queue (blocks free curr/low): hardware (255/0)

  Traffic Statistics for "Outside":

        533805632 packets input, 87594789839 bytes

        531836009 packets output, 514115438627 bytes

        8499758 packets dropped

      1 minute input rate 3201 pkts/sec,  456343 bytes/sec

      1 minute output rate 3414 pkts/sec,  3194657 bytes/sec

      1 minute drop rate, 42 pkts/sec

      5 minute input rate 3192 pkts/sec,  462319 bytes/sec

      5 minute output rate 3347 pkts/sec,  3087920 bytes/sec

      5 minute drop rate, 44 pkts/sec

Interface GigabitEthernet0/1 "", is up, line protocol is up

  Hardware is i82546GB rev03, BW 1000 Mbps, DLY 10 usec

        Auto-Duplex(Full-duplex), Auto-Speed(1000 Mbps)

        Input flow control is unsupported, output flow control is off

        Available but not configured via nameif

        MAC address 503d.e51e.ca07, MTU not set

        IP address unassigned

        52743008732 packets input, 10981273483975 bytes, 0 no buffer

        Received 67316769 broadcasts, 0 runts, 0 giants

        227203 input errors, 0 CRC, 0 frame, 227203 overrun, 0 ignored, 0 abort

        0 pause input, 0 resume input

        0 L2 decode drops

        52646387186 packets output, 10004093219462 bytes, 59370 underruns

        0 pause output, 0 resume output

        0 output errors, 0 collisions, 2 interface resets

        0 late collisions, 0 deferred

        0 input reset drops, 0 output reset drops, 0 tx hangs

        input queue (blocks free curr/low): hardware (255/230)

        output queue (blocks free curr/low): hardware (253/0)

Interface GigabitEthernet0/1.700 "xxxx", is up, line protocol is up

# Attention: This interface is located in a PCI-e x23 slot. For #

# optimal throughput, install the interface in a PCI-e x26 slot #

# if one is available. Refer to 'show controller slot'.        #

  Hardware is i82546GB rev03, BW 1000 Mbps, DLY 10 usec

        VLAN identifier 700

        Description: xxxx

        MAC address 503d.e51e.ca07, MTU 1500

        IP address xxxx, subnet mask 255.255.252.0

  Traffic Statistics for "xxxx":

        1484005458 packets input, 945109157423 bytes

        1549237026 packets output, 1000635823115 bytes

        8044767 packets dropped

Interface GigabitEthernet0/1.702 "xxxx", is up, line protocol is up

# Attention: This interface is located in a PCI-e x23 slot. For #

# optimal throughput, install the interface in a PCI-e x26 slot #

# if one is available. Refer to 'show controller slot'.        #

  Hardware is i82546GB rev03, BW 1000 Mbps, DLY 10 usec

        VLAN identifier 702

        Description: xxxx

        MAC address 503d.e51e.ca07, MTU 1500

        IP address xxxx, subnet mask 255.255.252.0

  Traffic Statistics for "xxxx":

        8943750366 packets input, 2755931796297 bytes

        8526491966 packets output, 1914193676672 bytes

        6630966 packets dropped

Interface GigabitEthernet0/1.704 "xxxx", is up, line protocol is up

# Attention: This interface is located in a PCI-e x23 slot. For #

# optimal throughput, install the interface in a PCI-e x26 slot #

# if one is available. Refer to 'show controller slot'.        #

  Hardware is i82546GB rev03, BW 1000 Mbps, DLY 10 usec

        VLAN identifier 704

        Description: xxxx

        MAC address 503d.e51e.ca07, MTU 1500

        IP address xxxx, subnet mask 255.255.252.0

  Traffic Statistics for "xxxx":

        7016462086 packets input, 1450478554512 bytes

        7065543596 packets output, 1479140251162 bytes

        2880196 packets dropped

Interface GigabitEthernet0/1.706 "xxxx", is up, line protocol is up

# Attention: This interface is located in a PCI-e x23 slot. For #

# optimal throughput, install the interface in a PCI-e x26 slot #

# if one is available. Refer to 'show controller slot'.        #

  Hardware is i82546GB rev03, BW 1000 Mbps, DLY 10 usec

        VLAN identifier 706

        Description: xxxx

        MAC address 503d.e51e.ca07, MTU 1500

        IP address xxxx, subnet mask 255.255.252.0

  Traffic Statistics for "xxxx":

        64416266 packets input, 7923320970 bytes

        38348092 packets output, 36506546909 bytes

        17763881 packets dropped

Interface GigabitEthernet0/1.708 "xxxx", is up, line protocol is up

# Attention: This interface is located in a PCI-e x23 slot. For #

# optimal throughput, install the interface in a PCI-e x26 slot #

# if one is available. Refer to 'show controller slot'.        #

  Hardware is i82546GB rev03, BW 1000 Mbps, DLY 10 usec

        VLAN identifier 708

        Description: xxxx

        MAC address 503d.e51e.ca07, MTU 1500

        IP address xxxx, subnet mask 255.255.252.0

  Traffic Statistics for "xxxx":

        499990620 packets input, 132317640462 bytes

        479568124 packets output, 94656034995 bytes

        5732396 packets dropped

Interface GigabitEthernet0/1.710 "xxxx", is up, line protocol is up

# Attention: This interface is located in a PCI-e x23 slot. For #

# optimal throughput, install the interface in a PCI-e x26 slot #

# if one is available. Refer to 'show controller slot'.        #

  Hardware is i82546GB rev03, BW 1000 Mbps, DLY 10 usec

        VLAN identifier 710

        Description: xxxx

        MAC address 503d.e51e.ca07, MTU 1500

        IP address xxxx, subnet mask 255.255.252.0

  Traffic Statistics for "xxxx":

        456697118 packets input, 142353101507 bytes

        436063268 packets output, 77074679645 bytes

        2048852 packets dropped

Interface GigabitEthernet0/1.712 "xxxx", is up, line protocol is up

# Attention: This interface is located in a PCI-e x23 slot. For #

# optimal throughput, install the interface in a PCI-e x26 slot #

# if one is available. Refer to 'show controller slot'.        #

  Hardware is i82546GB rev03, BW 1000 Mbps, DLY 10 usec

        VLAN identifier 712

        Description: xxxx

        MAC address 503d.e51e.ca07, MTU 1500

        IP address xxxx, subnet mask 255.255.252.0

  Traffic Statistics for "xxxx":

        4611219 packets input, 374277118 bytes

        3161754 packets output, 1164648244 bytes

        2043267 packets dropped

Interface GigabitEthernet0/1.714 "xxxx", is up, line protocol is up

# Attention: This interface is located in a PCI-e x23 slot. For #

# optimal throughput, install the interface in a PCI-e x26 slot #

# if one is available. Refer to 'show controller slot'.        #

  Hardware is i82546GB rev03, BW 1000 Mbps, DLY 10 usec

        VLAN identifier 714

        Description: xxxx

        MAC address 503d.e51e.ca07, MTU 1500

        IP address xxxx, subnet mask 255.255.252.0

  Traffic Statistics for "xxxx":

        32096378 packets input, 4467537888 bytes

        27557898 packets output, 16522329528 bytes

        4961549 packets dropped

Interface GigabitEthernet0/1.716 "xxxx", is up, line protocol is up

# Attention: This interface is located in a PCI-e x23 slot. For #

# optimal throughput, install the interface in a PCI-e x26 slot #

# if one is available. Refer to 'show controller slot'.        #

  Hardware is i82546GB rev03, BW 1000 Mbps, DLY 10 usec

        VLAN identifier 716

        Description: xxxx

        MAC address 503d.e51e.ca07, MTU 1500

        IP address xxxx, subnet mask 255.255.252.0

  Traffic Statistics for "xxxx":

        18831803 packets input, 8344011207 bytes

        22523531 packets output, 22066952655 bytes

        2861535 packets dropped

Interface GigabitEthernet0/1.718 "xxxx", is up, line protocol is up

# Attention: This interface is located in a PCI-e x23 slot. For #

# optimal throughput, install the interface in a PCI-e x26 slot #

# if one is available. Refer to 'show controller slot'.        #

  Hardware is i82546GB rev03, BW 1000 Mbps, DLY 10 usec

        VLAN identifier 718

        Description: xxxx

        MAC address 503d.e51e.ca07, MTU 1500

        IP address xxxx, subnet mask 255.255.252.0

  Traffic Statistics for "xxxx":

        989577333 packets input, 649122513712 bytes

        1178962688 packets output, 379217574278 bytes

        467267 packets dropped

Interface GigabitEthernet0/1.720 "xxxx", is up, line protocol is up

# Attention: This interface is located in a PCI-e x23 slot. For #

# optimal throughput, install the interface in a PCI-e x26 slot #

# if one is available. Refer to 'show controller slot'.        #

  Hardware is i82546GB rev03, BW 1000 Mbps, DLY 10 usec

        VLAN identifier 720

        Description: xxxx

        MAC address 503d.e51e.ca07, MTU 1500

        IP address xxxx, subnet mask 255.255.252.0

  Traffic Statistics for "xxxx":

        3 packets input, 126 bytes

        4 packets output, 112 bytes

        0 packets dropped

Interface GigabitEthernet0/1.722 "xxxx", is up, line protocol is up

# Attention: This interface is located in a PCI-e x23 slot. For #

# optimal throughput, install the interface in a PCI-e x26 slot #

# if one is available. Refer to 'show controller slot'.        #

  Hardware is i82546GB rev03, BW 1000 Mbps, DLY 10 usec

        VLAN identifier 722

        Description: xxxx

        MAC address 503d.e51e.ca07, MTU 1500

        IP address xxxx, subnet mask 255.255.252.0

  Traffic Statistics for "xxxx":

        284902407 packets input, 36413839768 bytes

        388643668 packets output, 140011990801 bytes

        5724539 packets dropped

Interface GigabitEthernet0/1.724 "xxxx", is up, line protocol is up

# Attention: This interface is located in a PCI-e x23 slot. For #

# optimal throughput, install the interface in a PCI-e x26 slot #

# if one is available. Refer to 'show controller slot'.        #

  Hardware is i82546GB rev03, BW 1000 Mbps, DLY 10 usec

        VLAN identifier 724

        Description: xxxx

        MAC address 503d.e51e.ca07, MTU 1500

        IP address xxxx, subnet mask 255.255.252.0

  Traffic Statistics for "xxxx":

        92120267 packets input, 80528735942 bytes

        97416524 packets output, 69199084337 bytes

        2229445 packets dropped

Interface GigabitEthernet0/1.726 "xxxx", is up, line protocol is up

# Attention: This interface is located in a PCI-e x23 slot. For #

# optimal throughput, install the interface in a PCI-e x26 slot #

# if one is available. Refer to 'show controller slot'.        #

  Hardware is i82546GB rev03, BW 1000 Mbps, DLY 10 usec

        VLAN identifier 726

        Description: xxxx

        MAC address 503d.e51e.ca07, MTU 1500

        IP address xxxx, subnet mask 255.255.252.0

  Traffic Statistics for "xxxx":

        3 packets input, 126 bytes

        4 packets output, 112 bytes

        0 packets dropped

Interface GigabitEthernet0/1.728 "xxxx", is up, line protocol is up

# Attention: This interface is located in a PCI-e x23 slot. For #

# optimal throughput, install the interface in a PCI-e x26 slot #

# if one is available. Refer to 'show controller slot'.        #

  Hardware is i82546GB rev03, BW 1000 Mbps, DLY 10 usec

        VLAN identifier 728

        Description: xxxx

        MAC address 503d.e51e.ca07, MTU 1500

        IP address xxxx, subnet mask 255.255.252.0

  Traffic Statistics for "xxxx":

        3 packets input, 126 bytes

        4 packets output, 112 bytes

        0 packets dropped

Interface GigabitEthernet0/1.730 "xxxx", is up, line protocol is up

# Attention: This interface is located in a PCI-e x23 slot. For #

# optimal throughput, install the interface in a PCI-e x26 slot #

# if one is available. Refer to 'show controller slot'.        #

  Hardware is i82546GB rev03, BW 1000 Mbps, DLY 10 usec

        VLAN identifier 730

        Description: xxxx

        MAC address 503d.e51e.ca07, MTU 1500

        IP address xxxx, subnet mask 255.255.252.0

  Traffic Statistics for "xxxx":

        3068433 packets input, 570069307 bytes

        2561350 packets output, 693274747 bytes

        826890 packets dropped

Interface GigabitEthernet0/1.732 "xxxx", is up, line protocol is up

# Attention: This interface is located in a PCI-e x23 slot. For #

# optimal throughput, install the interface in a PCI-e x26 slot #

# if one is available. Refer to 'show controller slot'.        #

  Hardware is i82546GB rev03, BW 1000 Mbps, DLY 10 usec

        VLAN identifier 732

        Description: xxxx

        MAC address 503d.e51e.ca07, MTU 1500

        IP address xxxx, subnet mask 255.255.252.0

  Traffic Statistics for "xxxx":

        8974460 packets input, 565390165 bytes

        32 packets output, 896 bytes

        8974413 packets dropped

Interface GigabitEthernet0/1.734 "xxxx", is up, line protocol is up

# Attention: This interface is located in a PCI-e x23 slot. For #

# optimal throughput, install the interface in a PCI-e x26 slot #

# if one is available. Refer to 'show controller slot'.        #

  Hardware is i82546GB rev03, BW 1000 Mbps, DLY 10 usec

        VLAN identifier 734

        Description: xxxx

        MAC address 503d.e51e.ca07, MTU 1500

        IP address xxxx, subnet mask 255.255.252.0

  Traffic Statistics for "xxxx":

        32835704482 packets input, 3603954525664 bytes

        32826034457 packets output, 3611196110356 bytes

        8986964 packets dropped

Interface GigabitEthernet0/1.736 "xxxx", is up, line protocol is up

# Attention: This interface is located in a PCI-e x23 slot. For #

# optimal throughput, install the interface in a PCI-e x26 slot #

# if one is available. Refer to 'show controller slot'.        #

  Hardware is i82546GB rev03, BW 1000 Mbps, DLY 10 usec

        VLAN identifier 736

        MAC address 503d.e51e.ca07, MTU 1500

        IP address xxxx, subnet mask 255.255.255.0

  Traffic Statistics for "xxxx":

        33 packets input, 1386 bytes

        33 packets output, 924 bytes

        0 packets dropped

Interface GigabitEthernet0/1.737 "xxxx", is up, line protocol is up

# Attention: This interface is located in a PCI-e x23 slot. For #

# optimal throughput, install the interface in a PCI-e x26 slot #

# if one is available. Refer to 'show controller slot'.        #

  Hardware is i82546GB rev03, BW 1000 Mbps, DLY 10 usec

        VLAN identifier 737

        MAC address 503d.e51e.ca07, MTU 1500

        IP address xxxx, subnet mask 255.255.255.0

  Traffic Statistics for "xxxx":

        1517009 packets input, 121262718 bytes

        751299 packets output, 325112414 bytes

        816845 packets dropped

Interface GigabitEthernet0/1.738 "xxxx", is up, line protocol is up

# Attention: This interface is located in a PCI-e x23 slot. For #

# optimal throughput, install the interface in a PCI-e x26 slot #

# if one is available. Refer to 'show controller slot'.        #

  Hardware is i82546GB rev03, BW 1000 Mbps, DLY 10 usec

        VLAN identifier 738

        MAC address 503d.e51e.ca07, MTU 1500

        IP address xxxx, subnet mask 255.255.255.0

  Traffic Statistics for "xxxx":

        5113485 packets input, 350891222 bytes

        3161353 packets output, 1932620453 bytes

        2453351 packets dropped

Interface GigabitEthernet0/1.739 "xxxx", is up, line protocol is up

# Attention: This interface is located in a PCI-e x23 slot. For #

# optimal throughput, install the interface in a PCI-e x26 slot #

# if one is available. Refer to 'show controller slot'.        #

  Hardware is i82546GB rev03, BW 1000 Mbps, DLY 10 usec

        VLAN identifier 739

        MAC address 503d.e51e.ca07, MTU 1500

        IP address xxxx, subnet mask 255.255.255.0

  Traffic Statistics for "xxxx":

        771783 packets input, 104804795 bytes

        431603 packets output, 305062467 bytes

        408596 packets dropped

Interface GigabitEthernet0/2 "", is administratively down, line protocol is down

  Hardware is i82546GB rev03, BW 1000 Mbps, DLY 10 usec

        Auto-Duplex, Auto-Speed

        Input flow control is unsupported, output flow control is off

        Available but not configured via nameif

        MAC address 503d.e51e.ca08, MTU not set

        IP address unassigned

        0 packets input, 0 bytes, 0 no buffer

        Received 0 broadcasts, 0 runts, 0 giants

        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

        0 pause input, 0 resume input

        0 L2 decode drops

        0 packets output, 0 bytes, 0 underruns

        0 pause output, 0 resume output

        0 output errors, 0 collisions, 2 interface resets

        0 late collisions, 0 deferred

        0 input reset drops, 0 output reset drops, 0 tx hangs

        input queue (blocks free curr/low): hardware (255/255)

        output queue (blocks free curr/low): hardware (255/255)

Interface GigabitEthernet0/3 "Failover", is up, line protocol is up

  Hardware is i82546GB rev03, BW 1000 Mbps, DLY 10 usec

        Auto-Duplex(Full-duplex), Auto-Speed(1000 Mbps)

        Input flow control is unsupported, output flow control is off

        Description: LAN/STATE Failover Interface

        MAC address 503d.e51e.ca09, MTU 1500

        IP address 10.255.255.1, subnet mask 255.255.255.252

        843173 packets input, 276753572 bytes, 0 no buffer

        Received 61 broadcasts, 0 runts, 0 giants

        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

        0 pause input, 0 resume input

        0 L2 decode drops

        577776501 packets output, 687842224236 bytes, 0 underruns

        0 pause output, 0 resume output

        0 output errors, 0 collisions, 1 interface resets

        0 late collisions, 0 deferred

        18 input reset drops, 0 output reset drops, 0 tx hangs

        input queue (blocks free curr/low): hardware (255/230)

        output queue (blocks free curr/low): hardware (228/178)

  Traffic Statistics for "Failover":

        843037 packets input, 257733014 bytes

        577776358 packets output, 677442239604 bytes

        0 packets dropped

      1 minute input rate 1 pkts/sec,  127 bytes/sec

      1 minute output rate 1579 pkts/sec,  1852671 bytes/sec

      1 minute drop rate, 0 pkts/sec

      5 minute input rate 1 pkts/sec,  125 bytes/sec

      5 minute output rate 1575 pkts/sec,  1848475 bytes/sec

      5 minute drop rate, 0 pkts/sec

Interface Management0/0 "management", is down, line protocol is down

  Hardware is i82557, BW 100 Mbps, DLY 100 usec

        Auto-Duplex, Auto-Speed

        Input flow control is unsupported, output flow control is unsupported

        MAC address 503d.e51e.ca05, MTU 1500

        IP address 192.168.1.1, subnet mask 255.255.255.0

        0 packets input, 0 bytes, 0 no buffer

        Received 0 broadcasts, 0 runts, 0 giants

        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

        0 pause input, 0 resume input

        0 L2 decode drops

        0 packets output, 0 bytes, 0 underruns

        0 pause output, 0 resume output

        0 output errors, 0 collisions, 0 interface resets

        0 babbles, 0 late collisions, 0 deferred

        0 lost carrier, 0 no carrier

        0 input reset drops, 0 output reset drops

        input queue (curr/max packets): hardware (0/0) software (0/0)

        output queue (curr/max packets): hardware (0/0) software (0/0)

  Traffic Statistics for "management":

        0 packets input, 0 bytes

        0 packets output, 0 bytes

        0 packets dropped

      1 minute input rate 0 pkts/sec,  0 bytes/sec

      1 minute output rate 0 pkts/sec,  0 bytes/sec

      1 minute drop rate, 0 pkts/sec

      5 minute input rate 0 pkts/sec,  0 bytes/sec

      5 minute output rate 0 pkts/sec,  0 bytes/sec

      5 minute drop rate, 0 pkts/sec

        Management-only interface. Blocked 0 through-the-device packets

Interface GigabitEthernet1/0 "", is administratively down, line protocol is down

  Hardware is VCS7380 rev01, BW 1000 Mbps, DLY 10 usec

        Auto-Duplex, Auto-Speed

        Input flow control is unsupported, output flow control is off

        Media-type configured as RJ45 connector

        Available but not configured via nameif

        MAC address 503d.e51e.ca1b, MTU not set

        IP address unassigned

        0 packets input, 0 bytes, 0 no buffer

        Received 0 broadcasts, 0 runts, 0 giants

        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

        0 pause input, 0 resume input

        0 L2 decode drops

        0 packets output, 0 bytes, 0 underruns

        0 pause output, 0 resume output

        0 output errors, 0 collisions, 0 interface resets

        0 late collisions, 0 deferred

        0 rate limit drops

        0 input reset drops, 0 output reset drops

        input queue (blocks free curr/low): hardware (0/0)

        output queue (blocks free curr/low): hardware (0/0)

Interface GigabitEthernet1/1 "", is administratively down, line protocol is down

  Hardware is VCS7380 rev01, BW 1000 Mbps, DLY 10 usec

        Auto-Duplex, Auto-Speed

        Input flow control is unsupported, output flow control is off

        Media-type configured as RJ45 connector

        Available but not configured via nameif

        MAC address 503d.e51e.ca1c, MTU not set

        IP address unassigned

        0 packets input, 0 bytes, 0 no buffer

        Received 0 broadcasts, 0 runts, 0 giants

        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

        0 pause input, 0 resume input

        0 L2 decode drops

        0 packets output, 0 bytes, 0 underruns

        0 pause output, 0 resume output

        0 output errors, 0 collisions, 0 interface resets

        0 late collisions, 0 deferred

        0 rate limit drops

        0 input reset drops, 0 output reset drops

        input queue (blocks free curr/low): hardware (0/0)

        output queue (blocks free curr/low): hardware (0/0)

Interface GigabitEthernet1/2 "", is administratively down, line protocol is down

  Hardware is VCS7380 rev01, BW 1000 Mbps, DLY 10 usec

        Auto-Duplex, Auto-Speed

        Input flow control is unsupported, output flow control is off

        Media-type configured as RJ45 connector

        Available but not configured via nameif

        MAC address 503d.e51e.ca1d, MTU not set

        IP address unassigned

        0 packets input, 0 bytes, 0 no buffer

        Received 0 broadcasts, 0 runts, 0 giants

        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

        0 pause input, 0 resume input

        0 L2 decode drops

        0 packets output, 0 bytes, 0 underruns

        0 pause output, 0 resume output

        0 output errors, 0 collisions, 0 interface resets

        0 late collisions, 0 deferred

        0 rate limit drops

        0 input reset drops, 0 output reset drops

        input queue (blocks free curr/low): hardware (0/0)

        output queue (blocks free curr/low): hardware (0/0)

Interface GigabitEthernet1/3 "", is administratively down, line protocol is down

  Hardware is VCS7380 rev01, BW 1000 Mbps, DLY 10 usec

        Auto-Duplex, Auto-Speed

        Input flow control is unsupported, output flow control is off

        Media-type configured as RJ45 connector

        Available but not configured via nameif

        MAC address 503d.e51e.ca1e, MTU not set

        IP address unassigned

        0 packets input, 0 bytes, 0 no buffer

        Received 0 broadcasts, 0 runts, 0 giants

        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

        0 pause input, 0 resume input

        0 L2 decode drops

        0 packets output, 0 bytes, 0 underruns

        0 pause output, 0 resume output

        0 output errors, 0 collisions, 0 interface resets

        0 late collisions, 0 deferred

        0 rate limit drops

        0 input reset drops, 0 output reset drops

        input queue (blocks free curr/low): hardware (0/0)

        output queue (blocks free curr/low): hardware (0/0)

Dispatch Unit - High CPU

Hello,

On fast ethernet 0/1 we can see a huge amount of overruns

  227203 input errors, 0 CRC, 0 frame, 227203 overrun, 0 ignored, 0 abort.

Now can you provide the show traffic to determine the troughput.

Regards,

Julio

Do rate helpful posts!

Julio Carvajal
Senior Network Security and Core Specialist
CCIE #42930, 2xCCNP, JNCIP-SEC
New Member

Dispatch Unit - High CPU

Hello, 

I don't mean to hi-Jack the thread but I'm having the same isses.  If I should start a new post please just say so.

I have a ASA5520 as a firewall, 300meg Internet connection and after upgrading from 7.x to 8x the cpu runs between 55% and 65%   Interfaces do not show any errors.

Cisco Adaptive Security Appliance Software Version 8.4(2)

Device Manager Version 6.4(5)

After clearing the inspect counters and waiting a couple of hours here is what I got.


Global policy:
  Service-policy: global_policy
    Class-map: inspection_default
      Inspect: dns preset_dns_map, packet 1486000, drop 209, reset-drop 0
      Inspect: ftp, packet 132, drop 0, reset-drop 0
      Inspect: h323 h225 _default_h323_map, packet 9, drop 0, reset-drop 0
               tcp-proxy: bytes in buffer 0, bytes dropped 0
      Inspect: h323 ras _default_h323_map, packet 1, drop 1, reset-drop 0
      Inspect: rsh, packet 0, drop 0, reset-drop 0
      Inspect: rtsp, packet 104508, drop 0, reset-drop 0
               tcp-proxy: bytes in buffer 0, bytes dropped 0
      Inspect: esmtp _default_esmtp_map, packet 24180, drop 0, reset-drop 0
      Inspect: sqlnet, packet 0, drop 0, reset-drop 0
      Inspect: skinny , packet 0, drop 0, reset-drop 0
               tcp-proxy: bytes in buffer 0, bytes dropped 0
      Inspect: sunrpc, packet 0, drop 0, reset-drop 0
               tcp-proxy: bytes in buffer 0, bytes dropped 0
      Inspect: xdmcp, packet 0, drop 0, reset-drop 0
      Inspect: sip , packet 230, drop 0, reset-drop 0
               tcp-proxy: bytes in buffer 0, bytes dropped 0
      Inspect: tftp, packet 0, drop 0, reset-drop 0
      Inspect: ip-options _default_ip_options_map, packet 0, drop 0, reset-drop 0
    Class-map: class-default

      Default Queueing  Packet recieved 0, sent 0, attack 0
Murray-ASA5520#

#

I removed the netbios inspect earlier, which had a large number of packets and I'm pretty sure I do not need it.  but wasn't sure if removing the dns or estmp inspects would help of cause problems because I do not fully understand what they do.

Thanks,

Tim

New Member

Dispatch Unit - High CPU

Tim,

All the better.  Looks like you are running the same version.  These high cpu issues can be tough to troubleshoot as I've found.

I think as long as it's relating to Dispatch unit and high cpu keep it here.

Have you taken a look at 'show asp drops'?

-E

New Member

Dispatch Unit - High CPU

Thanks !,  Here is the sh asp drop.  I don't understand yet what it means though. 

This is after being cleared and running about 2 hours.

Murray-ASA5520#  show asp drop

Frame drop:
  Invalid UDP Length (invalid-udp-length)                                      2
  Flow is denied by configured rule (acl-drop)                             773340
  First TCP packet not SYN (tcp-not-syn)                                   59940
  Bad TCP flags (bad-tcp-flags)                                                 402
  TCP Dual open denied (tcp-dual-open)                                     251
  TCP data send after FIN (tcp-data-past-fin)                               1
  TCP failed 3 way handshake (tcp-3whs-failed)                          3356
  TCP RST/FIN out of order (tcp-rstfin-ooo)                                 31048
  TCP SEQ in SYN/SYNACK invalid (tcp-seq-syn-diff)                124
  TCP SYNACK on established conn (tcp-synack-ooo)               59
  TCP packet SEQ past window (tcp-seq-past-win)                     11784
  TCP invalid ACK (tcp-invalid-ack)                                           41
  TCP RST/SYN in window (tcp-rst-syn-in-win)                                        250
  TCP packet failed PAWS test (tcp-paws-fail)                                       7574
  DNS Inspect invalid packet (inspect-dns-invalid-pak)                              5
  DNS Inspect invalid domain label (inspect-dns-invalid-domain-label)          8
  DNS Inspect packet too long (inspect-dns-pak-too-long)                          1
  DNS Inspect id not matched (inspect-dns-id-not-matched)                       92
  Dropped pending packets in a closed socket (np-socket-closed)               64

Last clearing: 14:44:29 EST Feb 1 2012 by hunter

Thanks !!

Tim

Dispatch Unit - High CPU

This is just going to tell us how much packets the ASA is dropping!.

We need to focuses first on

1- Is logging enabled

2- Amount of traffic traversing the ASA

3-Amount of connection per hosts on the ASA

Regards,

Julio

Do rate all the posts that help

Julio Carvajal
Senior Network Security and Core Specialist
CCIE #42930, 2xCCNP, JNCIP-SEC
New Member

Dispatch Unit - High CPU

Julio,

Thank you for your help.  I'm trying to catch the ASA during a period of high cpu.  That's been difficult.  Now my plan is to 'clear traffic', let it run for 24hrs, then issue the show traffic command.  I will post the findings tomorrow.

-E

Dispatch Unit - High CPU

Hello Edward,

Great, Lets keep this monitor.

I will be more than glad to help.

Regards,

Rate helpful posts

Julio Carvajal
Senior Network Security and Core Specialist
CCIE #42930, 2xCCNP, JNCIP-SEC
New Member

Dispatch Unit - High CPU

I am having the exact same problem after rebooting our ASA this past weekend, anxious to see what edward's result are.

Dispatch Unit - High CPU

Hello,

Are all having the high CPU with the Dispatch Unit on the top of the processes?

Regards,

Julio

Julio Carvajal
Senior Network Security and Core Specialist
CCIE #42930, 2xCCNP, JNCIP-SEC
New Member

Dispatch Unit - High CPU

Yes, here is my cpu usage.

01(config)# show processes cpu-usage

PC         Thread       5Sec     1Min     5Min   Process

08054f7c   c91afc90     0.0%     0.0%     0.0%   block_diag

081ab744   c91af8a0    83.9%    84.1%    85.0%   Dispatch Unit

083af4d5   c91af4b0     0.0%     0.0%     0.0%   CF OIR

08a43050   c91af2b8     0.0%     0.0%     0.0%   lina_int

08068a26   c91aecd0     0.0%     0.0%     0.0%   Reload Control Thread

08070c86   c91aead8     0.0%     0.0%     0.0%   aaa

08c53b1d   c91ae8e0     0.0%     0.0%     0.0%   UserFromCert Thread

08b80d6b   c91ae6e8     0.0%     0.0%     0.0%   Boot Message Proxy Process

080a1b36   c91ae4f0     0.0%     0.0%     0.0%   CMGR Server Process

080a2045   c91ae2f8     0.0%     0.0%     0.0%   CMGR Timer Process

081aab6c   c91ad920     0.0%     0.0%     0.0%   dbgtrace

084420fc   c91ad140     0.0%     0.0%     0.0%   557mcfix

08441f1e   c91acf48     0.0%     0.0%     0.0%   557statspoll

08c53b1d   c91abd90     0.0%     0.0%     0.0%   netfs_thread_init

092ae235   c91ab3b8     0.0%     0.0%     0.0%   Chunk Manager

088d047e   c91ab1c0     0.0%     0.0%     0.0%   PIX Garbage Collector

088c3904   c91aafc8     0.0%     0.0%     0.0%   IP Address Assign

08a92e56   c91aadd0     0.0%     0.0%     0.0%   QoS Support Module

0893faef   c91aabd8     0.0%     0.0%     0.0%   Client Update Task

092f8e9a   c91aa9e0     0.0%     0.0%     0.0%   Checkheaps

08a96945   c91aa3f8     0.0%     0.0%     0.0%   Quack process

08aedfd2   c91aa200     0.0%     0.0%     0.0%   Session Manager

08bffd55   c91a9e10     0.0%     0.0%     0.0%   uauth

08b9f655   c91a9c18     0.0%     0.0%     0.0%   Uauth_Proxy

08bd5d35   c91a9630     0.0%     0.0%     0.0%   SSL

08bfdce6   c91a9438     0.0%     0.0%     0.0%   SMTP

08bf68e6   c91a9240     0.0%     0.0%     0.0%   Logger

08bf7148   c91a9048     0.0%     0.0%     0.0%    Syslog Retry Thread

08bf117e   c91a8e50     0.0%     0.0%     0.0%   Thread Logger

08de42f2   c91a8088     0.0%     0.0%     0.0%   vpnlb_thread

08273dad   c91a78a8     0.0%     0.0%     0.0%   TLS Proxy Inspector

08b07c33   c91a76b0     0.0%     0.0%     0.0%   emweb/cifs_timer

08693087   c91a74b8     0.0%     0.0%     0.0%   netfs_mount_handler

08526b48   c91a72c0     0.0%     0.0%     0.0%   arp_timer

085306bc   c91a70c8     0.0%     0.0%     0.0%   arp_forward_thread

085a0925   c91a6ed0     0.0%     0.0%     0.0%   Lic TMR

08c02c61   c91a6cd8     0.0%     0.0%     0.0%   tcp_fast

08c05d31   c91a6ae0     0.0%     0.0%     0.0%   tcp_slow

08c31019   c91a68e8     0.0%     0.0%     0.0%   udp_timer

080feec8   c91a66f0     0.0%     0.0%     0.0%   CTCP Timer process

08d93793   c91a64f8     0.0%     0.0%     0.0%   L2TP data daemon

08d94563   c91a6300     0.0%     0.0%     0.0%   L2TP mgmt daemon

08d808f8   c91a6108     0.0%     0.0%     0.0%   ppp_timer_thread

08de47c7   c91a5f10     0.0%     0.0%     0.0%   vpnlb_timer_thread

0811581f   c91a5d18     0.0%     0.0%     0.0%   IPsec message handler

08128f5c   c91a5b20     0.0%     0.0%     0.0%   CTM message handler

089a16a9   c91a5928     0.0%     0.0%     0.0%   NAT security-level reconfiguration

08ac1eb8   c91a5730     0.0%     0.0%     0.0%   ICMP event handler

New Member

Dispatch Unit - High CPU

George,

I find 'show proc cpu-usage sorted non-zero' to be useful.  Gives you the highest on top and leaves out all the process with a zero value.

Just a thought

New Member

Dispatch Unit - High CPU

How do you use the nonzero part of the command, is it with a pipe?

New Member

Dispatch Unit - High CPU

George,

The command should be exactly as written 'show proc cpu-usage sorted non-zero'.  It may be dependent on version, not 100% sure.

-Eddie

32825
Views
35
Helpful
62
Replies
CreatePlease to create content