ASA 5520 High CPU

Unanswered Question
Feb 25th, 2010

Has anyone scene a problem with the ASA 5520 where the CPU runs very high?  We are sending 60Mbps of VOIP traffic through the devices and the CPU is close to 60%.  We plan on going as high as 150Mbps which would seem to be a disaster waiting to happen.

Thank you in advance for any help you can provide.

Matthew

Here is the output from show processes cpu-usage

vasa# show processes cpu-usage

PC         Thread       5Sec     1Min     5Min   Process

08054f7c   c91afc90     0.0%     0.0%     0.0%   block_diag

081ab744   c91af8a0    60.0%    57.6%    57.4%   Dispatch Unit

083af4d5   c91af4b0     0.0%     0.0%     0.0%   CF OIR

08a43050   c91af2b8     0.0%     0.0%     0.0%   lina_int

08068a26   c91aecd0     0.0%     0.0%     0.0%   Reload Control Thread

08070c86   c91aead8     0.0%     0.0%     0.0%   aaa

08c53b1d   c91ae8e0     0.0%     0.0%     0.0%   UserFromCert Thread

080a1b36   c91ae4f0     0.0%     0.0%     0.0%   CMGR Server Process

080a2045   c91ae2f8     0.0%     0.0%     0.0%   CMGR Timer Process

08c310c8   c91ae100     0.0%     0.0%     0.0%   snmp

081aab6c   c91ad920     0.0%     0.0%     0.0%   dbgtrace

I have this problem too.
1 vote
  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 4.2 (5 ratings)
Loading.
Joe B Danford Mon, 03/01/2010 - 13:01

Matthew,


We probably need to look at more than just show proc output for this. Can you get the following information?


show mem

show run threat

show interface

show service-policy global

cpu profile activate 18000 (wait 3 minutes)
show cpu profile dump
clear cpu profile dump
show conn count
sh xlate count
sh cpu usage


Thanks,

Joe
matthew_brown Mon, 04/05/2010 - 11:19

Joe,

Thank you for your reply and I am sorry I have not been able to as of yet.  I have some good news on the CPU usage as we were able to get it down a bit by working with Cisco TAC and by disabling the inspect services and some other minor items.  The answer that Cisco TAC gave us is that due to the small packet size of the VOIP traffic the ASA5520 firewall is not able to handle the throughput that the data sheet lists and that the CPU usage we see is what we should see.  You can see by the two below Cacti graphs what our CPU usage is in comparison to network usage.  The good news is that there is not something wrong.  The bad news is that we might have to upgrade the firewalls sooner rather than later.

Thank you again for your willingness to help.

Matthew

vasa - 5 Minute CPU Usage

vasa - Traffic - vlan650

ahmad82pkn Fri, 05/11/2012 - 08:50

we are facing same issue, can you confirm what inspectino you disabled?

Mokeev.tel Thu, 05/23/2013 - 01:34

Hi.

I have some issue. I see cpu profile is can help me. But, how to analyze it?

Joe B Danford Thu, 05/23/2013 - 05:10

Hi Sergey. Can you post the CPU profile output? The total command below would be useful also.

show mem

show run threat

show interface

show service-policy global

cpu profile activate 18000 (wait 3 minutes)
show cpu profile dump
clear cpu profile dump
show conn count
sh xlate count
sh cpu usage
Mokeev.tel Thu, 05/23/2013 - 05:21

Ok, here copy of terminal:

Type help or '?' for a list of available commands.

i-ASA-5520> en

Password: ***********

i-ASA-5520# show mem

Free memory:      1804649504 bytes (84%)

Used memory:       342834144 bytes (16%)

-------------     ----------------

Total memory:     2147483648 bytes (100%)

i-ASA-5520# show run threat

no threat-detection basic-threat

no threat-detection statistics access-list

no threat-detection statistics tcp-intercept

i-ASA-5520# term pa 0

i-ASA-5520# show interface

Interface GigabitEthernet0/0 "outside", is up, line protocol is up

  Hardware is i82546GB rev03, BW 1000 Mbps, DLY 10 usec

        Auto-Duplex(Full-duplex), Auto-Speed(1000 Mbps)

        Input flow control is unsupported, output flow control is off

        MAC address 0022.bdeb.0e36, MTU 1500

        IP address hidden, subnet mask 255.255.255.0

        64503535438 packets input, 12898043722199 bytes, 0 no buffer

        Received 28647836 broadcasts, 0 runts, 0 giants

        26872247 input errors, 0 CRC, 0 frame, 26872247 overrun, 0 ignored, 0 abort

        0 pause input, 0 resume input

        0 L2 decode drops

        60481707456 packets output, 12093844986224 bytes, 783465 underruns

        0 pause output, 0 resume output

        0 output errors, 0 collisions, 0 interface resets

        0 late collisions, 0 deferred

        0 input reset drops, 0 output reset drops, 0 tx hangs

        input queue (blocks free curr/low): hardware (255/230)

        output queue (blocks free curr/low): hardware (255/0)

  Traffic Statistics for "outside":

        64503531987 packets input, 11506268768313 bytes

        60482490921 packets output, 10893798552667 bytes

        28389542 packets dropped

      1 minute input rate 35776 pkts/sec,  8100147 bytes/sec

      1 minute output rate 31590 pkts/sec,  6985328 bytes/sec

      1 minute drop rate, 3 pkts/sec

      5 minute input rate 33197 pkts/sec,  6543319 bytes/sec

      5 minute output rate 30150 pkts/sec,  6253108 bytes/sec

      5 minute drop rate, 3 pkts/sec

Interface GigabitEthernet0/1 "inside", is up, line protocol is up

  Hardware is i82546GB rev03, BW 1000 Mbps, DLY 10 usec

        Auto-Duplex(Full-duplex), Auto-Speed(1000 Mbps)

        Input flow control is unsupported, output flow control is off

        MAC address 0022.bdeb.0e37, MTU 1500

        IP address 10.1.0.2, subnet mask 255.255.0.0

        62025158566 packets input, 12258925225796 bytes, 0 no buffer

        Received 5386 broadcasts, 0 runts, 0 giants

        31793228 input errors, 0 CRC, 0 frame, 31793228 overrun, 0 ignored, 0 abort

        0 pause input, 0 resume input

        0 L2 decode drops

        65987853474 packets output, 13047219797087 bytes, 12038 underruns

        0 pause output, 0 resume output

        0 output errors, 0 collisions, 7 interface resets

        0 late collisions, 0 deferred

        0 input reset drops, 56 output reset drops, 5 tx hangs

        input queue (blocks free curr/low): hardware (255/230)

        output queue (blocks free curr/low): hardware (255/0)

  Traffic Statistics for "inside":

        62025156768 packets input, 11023623340317 bytes

        65987866486 packets output, 11622260830857 bytes

        13691190 packets dropped

      1 minute input rate 32406 pkts/sec,  7142032 bytes/sec

      1 minute output rate 36588 pkts/sec,  8256627 bytes/sec

      1 minute drop rate, 1 pkts/sec

      5 minute input rate 30961 pkts/sec,  6409830 bytes/sec

      5 minute output rate 34003 pkts/sec,  6699719 bytes/sec

      5 minute drop rate, 1 pkts/sec

Interface GigabitEthernet0/2 "", is up, line protocol is up

  Hardware is i82546GB rev03, BW 1000 Mbps, DLY 10 usec

        Auto-Duplex(Full-duplex), Auto-Speed(1000 Mbps)

        Input flow control is unsupported, output flow control is off

        MAC address 0022.bdeb.0e92, MTU not set

        IP address unassigned

        3225550759 packets input, 3741995595916 bytes, 0 no buffer

        Received 3352 broadcasts, 0 runts, 0 giants

        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

        0 pause input, 0 resume input

        0 L2 decode drops

        603543170 packets output, 651564904056 bytes, 0 underruns

        0 pause output, 0 resume output

        0 output errors, 0 collisions, 3 interface resets

        0 late collisions, 0 deferred

        0 input reset drops, 0 output reset drops, 0 tx hangs

        input queue (blocks free curr/low): hardware (255/230)

        output queue (blocks free curr/low): hardware (255/151)

Interface GigabitEthernet0/2.1 "failover", is up, line protocol is up

# Attention: This interface is located in a PCI-e x0 slot. For #

# optimal throughput, install the interface in a PCI-e x60 slot #

# if one is available. Refer to 'show controller slot'.        #

  Hardware is i82546GB rev03, BW 1000 Mbps, DLY 10 usec

        VLAN identifier 123

        Description: LAN Failover Interface

        MAC address 0022.bdeb.0e92, MTU 1500

        IP address 10.11.12.2, subnet mask 255.255.255.252

  Traffic Statistics for "failover":

        32366708 packets input, 3993654992 bytes

        32368254 packets output, 3891136200 bytes

        0 packets dropped

Interface GigabitEthernet0/2.2 "state", is up, line protocol is up

# Attention: This interface is located in a PCI-e x0 slot. For #

# optimal throughput, install the interface in a PCI-e x60 slot #

# if one is available. Refer to 'show controller slot'.        #

  Hardware is i82546GB rev03, BW 1000 Mbps, DLY 10 usec

        VLAN identifier 124

        Description: STATE Failover Interface

        MAC address 0022.bdeb.0e92, MTU 1500

        IP address 10.12.11.2, subnet mask 255.255.255.252

  Traffic Statistics for "state":

        3193184056 packets input, 3603823090430 bytes

        571175036 packets output, 634395818576 bytes

        0 packets dropped

Interface GigabitEthernet0/3 "", is administratively down, line protocol is down

  Hardware is i82546GB rev03, BW 1000 Mbps, DLY 10 usec

        Auto-Duplex, Auto-Speed

        Input flow control is unsupported, output flow control is off

        Available but not configured via nameif

        MAC address 0022.bdeb.0e93, MTU not set

        IP address unassigned

        0 packets input, 0 bytes, 0 no buffer

        Received 0 broadcasts, 0 runts, 0 giants

        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

        0 pause input, 0 resume input

        0 L2 decode drops

        0 packets output, 0 bytes, 0 underruns

        0 pause output, 0 resume output

        0 output errors, 0 collisions, 0 interface resets

        0 late collisions, 0 deferred

        0 input reset drops, 0 output reset drops, 0 tx hangs

        input queue (blocks free curr/low): hardware (255/255)

        output queue (blocks free curr/low): hardware (255/255)

Interface Management0/0 "management", is administratively down, line protocol is up

  Hardware is i82557, BW 100 Mbps, DLY 100 usec

        Auto-Duplex(Full-duplex), Auto-Speed(100 Mbps)

        Input flow control is unsupported, output flow control is unsupported

        MAC address 0022.bdeb.0e35, MTU 1500

        IP address 192.168.168.2, subnet mask 255.255.255.0

        150421 packets input, 9032160 bytes, 0 no buffer

        Received 150536 broadcasts, 0 runts, 0 giants

        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

        0 pause input, 0 resume input

        0 L2 decode drops

        398432 packets output, 25792376 bytes, 0 underruns

        0 pause output, 0 resume output

        0 output errors, 0 collisions, 0 interface resets

        0 babbles, 0 late collisions, 0 deferred

        0 lost carrier, 0 no carrier

        0 input reset drops, 0 output reset drops

        input queue (curr/max packets): hardware (0/1) software (0/11)

        output queue (curr/max packets): hardware (1/4) software (0/3)

  Traffic Statistics for "management":

        150536 packets input, 6924656 bytes

        398901 packets output, 11685908 bytes

        0 packets dropped

      1 minute input rate 0 pkts/sec,  0 bytes/sec

      1 minute output rate 0 pkts/sec,  0 bytes/sec

      1 minute drop rate, 0 pkts/sec

      5 minute input rate 0 pkts/sec,  0 bytes/sec

      5 minute output rate 0 pkts/sec,  0 bytes/sec

      5 minute drop rate, 0 pkts/sec

i-ASA-5520# show service-policy global

Global policy:

  Service-policy: global_policy

    Class-map: inspection_default

      Inspect: ftp, packet 287354, drop 0, reset-drop 0

      Inspect: h323 h225 _default_h323_map, packet 0, drop 0, reset-drop 0

               tcp-proxy: bytes in buffer 0, bytes dropped 0

      Inspect: h323 ras _default_h323_map, packet 0, drop 0, reset-drop 0

      Inspect: rsh, packet 0, drop 0, reset-drop 0

      Inspect: rtsp, packet 0, drop 0, reset-drop 0

               tcp-proxy: bytes in buffer 0, bytes dropped 0

      Inspect: esmtp _default_esmtp_map, packet 9844112, drop 27, reset-drop 0

      Inspect: sqlnet, packet 0, drop 0, reset-drop 0

      Inspect: xdmcp, packet 0, drop 0, reset-drop 0

      Inspect: tftp, packet 0, drop 0, reset-drop 0

      Inspect: ip-options _default_ip_options_map, packet 0, drop 0, reset-drop 0

      Inspect: dns migrated_dns_map_1, packet 101963293, drop 10147, reset-drop 0

      Inspect: icmp, packet 6754749, drop 497, reset-drop 0

      Inspect: sunrpc, packet 0, drop 0, reset-drop 0

               tcp-proxy: bytes in buffer 0, bytes dropped 0

      Inspect: sip , packet 0, drop 0, reset-drop 0

               tcp-proxy: bytes in buffer 0, bytes dropped 0

      Inspect: netbios, packet 46776, drop 0, reset-drop 0

i-ASA-5520# cpu profile activate 18000

Activated CPU profiling for 18000 samples.

Use "show cpu profile" to see the results.

i-ASA-5520# show cpu profile dump

Cisco Adaptive Security Appliance Software Version 8.2(5)

Hardware:   ASA5520

CPU profiling started: 09:07:33.799 EDT Thu May 23 2013

cores: 1

Process virtual address map:

---------------------------

08048000-098d4000 r-xp 00000000 00:01 332        /asa/bin/lina

098d4000-09d2d000 rwxp 0188c000 00:01 332        /asa/bin/lina

09d2d000-0a1d6000 rwxp 09d2d000 00:00 0          [heap]

6651f000-66520000 rwxp 6651f000 00:00 0

66522000-66523000 rwxp 66522000 00:00 0

66525000-66526000 rwxp 66525000 00:00 0

66627000-66628000 rwxp 66627000 00:00 0

66729000-6672a000 rwxp 66729000 00:00 0

6682b000-6682c000 rwxp 6682b000 00:00 0

6682e000-6682f000 rwxp 6682e000 00:00 0

66831000-66832000 rwxp 66831000 00:00 0

66834000-66835000 rwxp 66834000 00:00 0

66837000-66838000 rwxp 66837000 00:00 0

6683a000-6683b000 rwxp 6683a000 00:00 0

6683d000-6683e000 rwxp 6683d000 00:00 0

66840000-66841000 rwxp 66840000 00:00 0

66843000-66844000 rwxp 66843000 00:00 0

66846000-66847000 rwxp 66846000 00:00 0

66849000-6684a000 rwxp 66849000 00:00 0

6684c000-6684d000 rwxp 6684c000 00:00 0

6684f000-66850000 rwxp 6684f000 00:00 0

66852000-66853000 rwxp 66852000 00:00 0

66855000-66856000 rwxp 66855000 00:00 0

66858000-66859000 rwxp 66858000 00:00 0

6685b000-6685c000 rwxp 6685b000 00:00 0

6685e000-6685f000 rwxp 6685e000 00:00 0

66861000-66862000 rwxp 66861000 00:00 0

66864000-66865000 rwxp 66864000 00:00 0

66867000-66868000 rwxp 66867000 00:00 0

6686a000-6686b000 rwxp 6686a000 00:00 0

6686d000-6686e000 rwxp 6686d000 00:00 0

66870000-66871000 rwxp 66870000 00:00 0

66873000-66874000 rwxp 66873000 00:00 0

66876000-66877000 rwxp 66876000 00:00 0

66879000-6687a000 rwxp 66879000 00:00 0

6687c000-6687d000 rwxp 6687c000 00:00 0

6687f000-66880000 rwxp 6687f000 00:00 0

66882000-66883000 rwxp 66882000 00:00 0

66885000-66886000 rwxp 66885000 00:00 0

66888000-66889000 rwxp 66888000 00:00 0

6688b000-6688c000 rwxp 6688b000 00:00 0

6688e000-6688f000 rwxp 6688e000 00:00 0

66891000-66892000 rwxp 66891000 00:00 0

66894000-66895000 rwxp 66894000 00:00 0

66897000-66898000 rwxp 66897000 00:00 0

6689a000-6689b000 rwxp 6689a000 00:00 0

6689d000-6689e000 rwxp 6689d000 00:00 0

668a0000-668a1000 rwxp 668a0000 00:00 0

668a3000-668a4000 rwxp 668a3000 00:00 0

668a6000-668a7000 rwxp 668a6000 00:00 0

668a9000-668aa000 rwxp 668a9000 00:00 0

668ac000-668ad000 rwxp 668ac000 00:00 0

668af000-668b0000 rwxp 668af000 00:00 0

668b2000-668b3000 rwxp 668b2000 00:00 0

668b5000-668b6000 rwxp 668b5000 00:00 0

668b8000-668b9000 rwxp 668b8000 00:00 0

668bb000-668bc000 rwxp 668bb000 00:00 0

668be000-668bf000 rwxp 668be000 00:00 0

668c1000-668c2000 rwxp 668c1000 00:00 0

668c4000-668c5000 rwxp 668c4000 00:00 0

668c7000-668c8000 rwxp 668c7000 00:00 0

668ca000-668cb000 rwxp 668ca000 00:00 0

668cd000-668ce000 rwxp 668cd000 00:00 0

668d0000-668d1000 rwxp 668d0000 00:00 0

668d3000-668d4000 rwxp 668d3000 00:00 0

668d6000-668d7000 rwxp 668d6000 00:00 0

668d9000-668da000 rwxp 668d9000 00:00 0

668dc000-668dd000 rwxp 668dc000 00:00 0

668df000-668e0000 rwxp 668df000 00:00 0

668e2000-668e3000 rwxp 668e2000 00:00 0

668e5000-668e6000 rwxp 668e5000 00:00 0

668e8000-668e9000 rwxp 668e8000 00:00 0

668eb000-668ec000 rwxp 668eb000 00:00 0

668ee000-668ef000 rwxp 668ee000 00:00 0

668f1000-668f2000 rwxp 668f1000 00:00 0

668f4000-668f5000 rwxp 668f4000 00:00 0

668f7000-668f8000 rwxp 668f7000 00:00 0

668fa000-668fb000 rwxp 668fa000 00:00 0

668fd000-668fe000 rwxp 668fd000 00:00 0

66900000-66901000 rwxp 66900000 00:00 0

66903000-66904000 rwxp 66903000 00:00 0

66906000-66907000 rwxp 66906000 00:00 0

66909000-6690a000 rwxp 66909000 00:00 0

6690c000-6690d000 rwxp 6690c000 00:00 0

6690f000-66910000 rwxp 6690f000 00:00 0

66912000-66913000 rwxp 66912000 00:00 0

66915000-66916000 rwxp 66915000 00:00 0

66918000-66919000 rwxp 66918000 00:00 0

6691b000-6691c000 rwxp 6691b000 00:00 0

6691e000-6691f000 rwxp 6691e000 00:00 0

66921000-66922000 rwxp 66921000 00:00 0

66924000-66925000 rwxp 66924000 00:00 0

66927000-66928000 rwxp 66927000 00:00 0

6692a000-6692b000 rwxp 6692a000 00:00 0

6692d000-6692e000 rwxp 6692d000 00:00 0

66930000-66931000 rwxp 66930000 00:00 0

66933000-66934000 rwxp 66933000 00:00 0

66936000-66937000 rwxp 66936000 00:00 0

66939000-6693a000 rwxp 66939000 00:00 0

6693c000-6693d000 rwxp 6693c000 00:00 0

6693f000-66940000 rwxp 6693f000 00:00 0

66942000-66943000 rwxp 66942000 00:00 0

66945000-66946000 rwxp 66945000 00:00 0

66948000-66949000 rwxp 66948000 00:00 0

6694b000-6694c000 rwxp 6694b000 00:00 0

6694e000-6694f000 rwxp 6694e000 00:00 0

66951000-66952000 rwxp 66951000 00:00 0

66954000-66955000 rwxp 66954000 00:00 0

66957000-66958000 rwxp 66957000 00:00 0

6695a000-6695b000 rwxp 6695a000 00:00 0

6695d000-6695e000 rwxp 6695d000 00:00 0

66960000-66961000 rwxp 66960000 00:00 0

66963000-66964000 rwxp 66963000 00:00 0

66966000-66967000 rwxp 66966000 00:00 0

66969000-6696a000 rwxp 66969000 00:00 0

6696c000-6696d000 rwxp 6696c000 00:00 0

6696f000-66970000 rwxp 6696f000 00:00 0

66972000-66973000 rwxp 66972000 00:00 0

66975000-66976000 rwxp 66975000 00:00 0

66978000-66979000 rwxp 66978000 00:00 0

6697b000-6697c000 rwxp 6697b000 00:00 0

6697e000-6697f000 rwxp 6697e000 00:00 0

66981000-66982000 rwxp 66981000 00:00 0

66984000-66985000 rwxp 66984000 00:00 0

66987000-66988000 rwxp 66987000 00:00 0

6698a000-6698b000 rwxp 6698a000 00:00 0

6698d000-6698e000 rwxp 6698d000 00:00 0

66990000-66991000 rwxp 66990000 00:00 0

66993000-66994000 rwxp 66993000 00:00 0

66996000-66997000 rwxp 66996000 00:00 0

66999000-6699a000 rwxp 66999000 00:00 0

6699c000-6699d000 rwxp 6699c000 00:00 0

6699f000-669a0000 rwxp 6699f000 00:00 0

669a2000-669a3000 rwxp 669a2000 00:00 0

669a5000-669a6000 rwxp 669a5000 00:00 0

669a8000-669a9000 rwxp 669a8000 00:00 0

669ab000-669ac000 rwxp 669ab000 00:00 0

669ae000-669af000 rwxp 669ae000 00:00 0

669b1000-669b2000 rwxp 669b1000 00:00 0

669b4000-669b5000 rwxp 669b4000 00:00 0

669b7000-669b8000 rwxp 669b7000 00:00 0

669ba000-669bb000 rwxp 669ba000 00:00 0

669bd000-669be000 rwxp 669bd000 00:00 0

669c0000-669c1000 rwxp 669c0000 00:00 0

669c3000-669c4000 rwxp 669c3000 00:00 0

669c6000-669c7000 rwxp 669c6000 00:00 0

669c9000-669ca000 rwxp 669c9000 00:00 0

669cc000-669cd000 rwxp 669cc000 00:00 0

669cf000-669d0000 rwxp 669cf000 00:00 0

669d2000-669d3000 rwxp 669d2000 00:00 0

669d5000-669d6000 rwxp 669d5000 00:00 0

669d8000-669d9000 rwxp 669d8000 00:00 0

669db000-669dc000 rwxp 669db000 00:00 0

669de000-669df000 rwxp 669de000 00:00 0

669e1000-669e2000 rwxp 669e1000 00:00 0

669e4000-669e5000 rwxp 669e4000 00:00 0

669e7000-669e8000 rwxp 669e7000 00:00 0

669ea000-669eb000 rwxp 669ea000 00:00 0

669ed000-669ee000 rwxp 669ed000 00:00 0

669f0000-669f1000 rwxp 669f0000 00:00 0

669f3000-669f4000 rwxp 669f3000 00:00 0

669f6000-669f7000 rwxp 669f6000 00:00 0

669f9000-669fa000 rwxp 669f9000 00:00 0

669fc000-66c00000 rwxp 669fc000 00:00 0

66c00000-6bd68000 rwxs 00000000 00:01 1027       /dev/udma0

6bdfc000-6c000000 rwxp 6bdfc000 00:00 0

6c000000-6e400000 rwxs 00000000 00:0a 0          /SYSV00000002 (deleted)

6e400000-6e800000 r-xs 02400000 00:0a 0          /SYSV00000002 (deleted)

6e800000-6ec00000 rwxs 02800000 00:0a 0          /SYSV00000002 (deleted)

6ec00000-6f000000 r-xs 02c00000 00:0a 0          /SYSV00000002 (deleted)

6f000000-6f400000 r-xs 03000000 00:0a 0          /SYSV00000002 (deleted)

6f400000-6f800000 r-xs 03400000 00:0a 0          /SYSV00000002 (deleted)

6f800000-6fc00000 r-xs 03800000 00:0a 0          /SYSV00000002 (deleted)

6fc00000-70000000 r-xs 03c00000 00:0a 0          /SYSV00000002 (deleted)

70000000-70400000 r-xs 04000000 00:0a 0          /SYSV00000002 (deleted)

70400000-70800000 r-xs 04400000 00:0a 0          /SYSV00000002 (deleted)

70800000-70c00000 r-xs 04800000 00:0a 0          /SYSV00000002 (deleted)

70c00000-dd400000 rwxs 04c00000 00:0a 0          /SYSV00000002 (deleted)

dd493000-dd494000 rwxp dd493000 00:00 0

dd494000-dd495000 rwxs fff00000 00:01 516        /dev/mem

dd495000-dd499000 rwxs 00000000 00:07 1183       /dev/zero (deleted)

dd499000-dd49a000 rwxp dd499000 00:00 0

dd49a000-dd4ba000 rwxs f8100000 00:01 516        /dev/mem

dd4ba000-dd4da000 rwxs f8200000 00:01 516        /dev/mem

dd4da000-dd4fa000 rwxs f8220000 00:01 516        /dev/mem

dd4fa000-dd51a000 rwxs f8240000 00:01 516        /dev/mem

dd51a000-dd53a000 rwxs f8260000 00:01 516        /dev/mem

dd53a000-dd53b000 rwxs ffb00000 00:01 516        /dev/mem

dd53b000-dd54b000 rwxs fff00000 00:01 516        /dev/mem

dd54b000-dd553000 rwxs 000d8000 00:01 516        /dev/mem

dd553000-dd554000 rwxs f8340000 00:01 516        /dev/mem

dd554000-dd555000 rwxs f8300000 00:01 516        /dev/mem

dd555000-dd556000 rwxp dd555000 00:00 0

dd556000-dd66a000 r-xp 00000000 00:01 214        /lib/libc-2.3.2.so

dd66a000-dd66e000 rwxp 00114000 00:01 214        /lib/libc-2.3.2.so

dd66e000-dd671000 rwxp dd66e000 00:00 0

dd671000-dd67b000 r-xp 00000000 00:01 179        /lib/libgcc_s.so.1

dd67b000-dd67c000 rwxp 00009000 00:01 179        /lib/libgcc_s.so.1

dd67c000-dd69b000 r-xp 00000000 00:01 186        /lib/libm-2.3.2.so

dd69b000-dd69c000 rwxp 0001e000 00:01 186        /lib/libm-2.3.2.so

dd69c000-dd69e000 r-xp 00000000 00:01 216        /lib/libdl-2.3.2.so

dd69e000-dd69f000 rwxp 00001000 00:01 216        /lib/libdl-2.3.2.so

dd69f000-dd6a0000 rwxp dd69f000 00:00 0

dd6a0000-dd6ac000 r-xp 00000000 00:01 183        /lib/libpthread-0.10.so

dd6ac000-dd6ad000 rwxp 0000c000 00:01 183        /lib/libpthread-0.10.so

dd6ad000-dd6ef000 rwxp dd6ad000 00:00 0

dd6ef000-dd709000 r-xp 00000000 00:01 315        /usr/lib/libexpat.so.1.5.2

dd709000-dd70b000 rwxp 00019000 00:01 315        /usr/lib/libexpat.so.1.5.2

dd70b000-dd7db000 r-xp 00000000 00:01 226        /lib/libstdc++.so.6.0.6

dd7db000-dd7e0000 rwxp 000d0000 00:01 226        /lib/libstdc++.so.6.0.6

dd7e0000-dd7e5000 rwxp dd7e0000 00:00 0

dd7e5000-dd7ea000 r-xp 00000000 00:01 311        /usr/lib/libnuma.so.1

dd7ea000-dd7eb000 rwxp 00004000 00:01 311        /usr/lib/libnuma.so.1

dd7eb000-dd7ed000 rwxp dd7eb000 00:00 0

dd7ed000-dd7ff000 r-xp 00000000 00:01 221        /lib/ld-2.3.2.so

dd7ff000-dd800000 rwxp 00011000 00:01 221        /lib/ld-2.3.2.so

ddfeb000-de000000 rwxp ddfeb000 00:00 0          [stack]

ffffe000-fffff000 r-xp 00000000 00:00 0          [vdso]

---------------------------

End of process map

Samples for core 0 - stopped

085d159e 081a872e 08442c74 08195289 085d2184 08717ecc 08189334 0933b9d9

085d1f30 081945a0 08064cb3 081aa54f 08716eb7 08442ab1 08717bf9 08716e5e

08451421 0817fe81 08064366 084419e5 08445b0d 081ab90c 0843f6f0 08717bd5

08c3b190 dd60ec47 08acf40a 085d159e 0871890f 0874d2ed 0804eb18 08c3aa08

081a87ab 0844017a 08440167 08c4f688 08064353 0804eb2d dd60ec47 0817fe36

08065be0 09345f00 0817fc60 086c968a 08717351 080642aa 081892e0 085d1680

08172ecb 08172eae 0819460e 08064394 08440193 0936474c 081aa558 0843f6e3

dd60ec47 087510f1 0818ea9b 0844019f 0804eabd 0875ff51 0844dffb 08c4fa21

08716f67 08c50015 08445a5f 0843f827 0844dffb 08a5edd6 0819463d 0843f6d5

0871bddd 08172ec9 08716f67 08195fb4 08c3b190 0843f6e9 0805208e dd6a918e

081a872a 08445b79 0844e511 081ab926 08445a36 081a872e 086c50a5 08442afe

0816ffd8 080600a4 08172eae 086debb3 0875b8e3 08172ec9 08c4f602 08716e8d

08c39387 081ab1bf 08716e78 08445aeb 09346c3a 08170d82 08717333 0843fb90

081735ff 08a5edb5 081aa860 0843f6e3 08441976 087171a7 08c4f653 081a894a

08065c14 081aa534 08052169 08700242 086df389 08c4f67f 0804d670 08716e65

080642d5 0870235c 083fcffc 0875b467 08065d29 080eb180 08442bf9 08055aa2

08737dea 08717bd5 08172eae 086c87e8 081a874d 0817d900 0805220a 0817d9bb

081a8925 08714c08 081a8683 0822b2e6 08717e72 085d159e 0805210e 0806444a

0875991e 08064af5 0871aca1 0871bf83 08445a4d 0871c710 0804eabd 08442bf9

081a87fd 081ab91a 0819460e 0871bbe2 0817fe30 08c35c40 086e0ebd 08194c79

0844dffb 087000e0 081ab90c 0933b8d0 08717e7e 085d2184 081a8901 08052051

0875991e 08717ccc 087188fa 08440172 084419e5 08445a59 0843f6d2 dd6a918e

08055a97 08c35ee9 0843f6e9 08c3aa85 dd60ec47 081945d7 dd60ec47 086e1151

08760e16 0843f6eb 0871bcd4 087692a8 081944c8 08768cb1 08445a5f 08717ed9

08717bd5 08c3629c 08445af0 08715b4f 08172ec9 087682a0 08c4f9f6 08172e9e

08440156 0817fce8 081a87c1 08442185 08194cc5 08445a45 08c36021 086e65db

081a86d0 08c38a14 08172eea 0870058c 081892ea 086e66eb 0817fc60 08450281

08052188 08170de5 dd60ec47 08440156 087145b4 08716f10 08441e3e 08c500bb

08717358 08172ec9 080643d7 0817d98f 08060109 08c4f653 08717e80 0843f6d2

dd60ec47 08174249 08716f67 08714bb8 08445af0 081a8683 08c362a6 08717b82

0822b244 09343afb 08717e30 0871726c 086ff9fe 08700dae dd6a92dc 081a8922

dd60ec40 08064350 081a892e 08052266 08442ad3 08064a7c 08170d7f 081aa48f

08c381f1 08c35100 086c9070 08064246 08172ec9 080642b1 0874d2ed 08c50097

085d2100 08c4ff03 08172eae 081ab900 dd60ec40 08a6bb7e 0804c214 081a892b

085d2136 0843f6dc 081a872a 0871655a 08716f67 080521cd 08c4f9f6 08717bd8

08442b45 085d159e dd60ec47 081a86cb 08445a48 081ab913 0805aa94 dd60ec47

08172ec9 087166a4 08064441 081a872e 0936474c 081a872e 086ffe5e 0817fd29

085d2184 0844018a 08064af9 081a86d7 08194341 080600d6 0844dffb 08737d6f

08715b5f 081ab926 0875b9f0 08715b6f 08445a30 08716854 08065bf2 08183868

0806486a 0844dffb 081ab901 08445abb dd60ec47 08750124 08445a65 08715d7f

084401bd 08064a30 08445a65 0804eab0 08718975 0817d973 0933b97c 080eb180

080642b1 09345b10 086e68d1 08170ffa 0871656e 08065be0 086c9bf1 086de880

08715c70 08172ec9 0805aa94 08445ac1 dd6a9389 08170d90 08714c0e 08717bd8

086c9066 0843f6c0 0871c726 08064246 0816ff50 08c375bb 080600bd 081838c1

dd60ec47 086c9066 08052036 dd60ec47 0817d9bb 086fff57 0818ea9e 0870060a

08195289 0817d9a8 08c3b057 081a87f0 08c35d05 081ab925 086ffe5e 0804eb1e

086e0d90 08195289 08172dff 08445a48 0817d901 08194afe 0804cc8f 081892f7

081892e1 080eb18c 08440150 08172ec9 08441a98 08171030 08764da3 0805aab1

08c375bb 081aace0 0844dffb 08c4fac5 0817107f 080522fb 08195196 081a8683

086ffe7d 08a5eda1 093472f3 08c4f624 dd6a6749 dd60ec47 dd60ec47 dd60ec47

086c3301 081aa860 08737d6f 0933b89a dd60ec47 0843f6e3 08172eae 08c35b03

0805aa80 08052207 dd6a92ce 08442272 086fffe0 087023a8 dd60ec47 080642d5

0805231a 081a8aa7 08c36282 08445a30 0875bcc1 084502a0 084401a9 086c9bf1

08c4f602 08acf3ff 087180e6 086d15a1 08445b0d 0805aab1 0843f6c0 08064d70

086b8b14 080eb1bd 08442187 0822b220 081ab189 dd60ec47 08715d93 08441d83

0818ea9e 081a8925 08716eac 08721f00 08064221 0818ea9e 0876616b 08170d7f

08c500bb 08716e8d 081a872e 081a872a 08716e96 086d20ff dd60ec47 086ffe5e

081aa58a 081a88ea 0817d947 08a6bbc0 0844dffb 08064a9d 0817101d 086c9066

08716e8d 087025c0 0890ba7c 0844dffb dd6a92ab 0844e2de 0843f6d5 081aa558

0818ea9e 0933b900 081945d7 0844dda1 0844dffb 0871be30 0934738d 086c41cd

08767682 081a8683 081a8726 08717333 087c7eb0 08055a97 08716f86 08716e8d

08716680 087510f0 08715c7f 08c373ff 081943cd 08441a83 08065be0 081aa532

08183953 09345c07 086debb3 0844dffb 081a8683 08715b7b 08c501cb 0871655d

086c88d4 08170ffa 08052211 08718901 086c36d8 08c358cf 08441a98 0818a380

0844e509 08064221 087145bf 081892f4 085d162b 08702384 081749e1 08c4ffa7

087b518e 081a87ce 086cca7e 081a8942 08064a58 081aa563 08c356b0 dd60ec47

081aace4 08764dc9 08717b7f 081aa548 08715c7f 0817fcb1 08715d37 08445a48

081aa460 0804cc8f 0817fe81 08172e9e 0871bba0 081aa492 081aa558 086ffe5e

0933b8e3 08c4f6a0 081a87e7 084401a2 0936474c 08445a30 087b8954 08c4f6a0

08c35c37 08a6bb7e 081ab1c9 080642b1 08177e5f 08718b85 0874e7bd 08445a70

0871bbf3 08172eae 08767f77 0819db29 08064246 08170f01 08445a65 dd60ec47

086de780 dd60ec47 08c39d8f 086c9076 08174287 0871be6c 080eb1bd 08170de5

0816ffb9 081a8687 0871bcad 081ab901 0817d9bb 086c5c18 0816ff79 08064af7

081a872e 087753b0 081aa558 08c363a0 085d2101 080600c5 0875c0ff 081a8732

08c4f67f 081a86d7 dd6a690e 0843f83a 08064246 086debb3 08715bc9 0871bfa0

dd60ec47 08716621 080600b8 081a87c5 dd6aa66f 08717bd5 08172eae 08768c02

0805aa9a 080600bd 08716571 0843f6d2 08440185 0817d9bb 08065be1 08442afb

0843f6c0 0844dffb 080642b1 084401db 08194387 dd60ec47 08c399fb 09348550

081a871b 08440196 080642aa 0817d9bb 081aa460 0933b9d9 0804ddf8 086debb3

dd60ec47 dd60ec47 08717e31 0875cf00 080642d5 08183ef9 0817d954 08c39d8f

086ffe87 0875faf5 08718991 08716571 081a872a 08716f67 08716f67 08c34cc0

08445b71 08c34d41 086d1887 0819460e 0804cc8f 08717bdf 086c9bf1 081a87ad

08716683 0817106a 08c35c37 0818a380 08a6bb81 0843f6e3 081aa479 086e66c3

0936474c 081a8687 08717330 087596ee 08441975 0805aaa0 0871c717 08194bca

08172eae 0933b96d 08450280 0817107f 081a872a 08a6bbc9 dd6a690e 0933b900

087648e5 dd60ec47 0875b5c2 086debb3 0843f6d2 08c3697f 081a88e3 081a873f

081ab900 08717be5 087185d9 080600cb 0805aa94 085d1ec4 08714eb0 0817fef2

08718b85 081aa555 08c39223 0843f6e3 0805219c 0817d9bb 0844018a 0804cc8f

080521aa dd60ec47 08445ad7 08c4f71c 08064c6b 0933b8f4 08442ad3 08195289

dd6a68f4 0817d9bb 08721f00 081aa466 dd60ec47 086deed6 08065be0 081a8901

08716f67 0817fe31 dd60ec47 0806456c 081a8683 081a8772 086e65d0 0843f6c6

0817fd2e 08714774 08717f1b 0843f6d2 08445b02 081ab901 08a630f2 0817fef2

081a87ce 08715d3f 081aa532 08194ccd 08172e9a 081a8922 08714580 0843f827

0875b618 08717ece 0843f6dc 081a8920 0844dffb 0875bcc1 08445a48 08064394

08716f28 08c4f688 0933b8f4 0843f6e3 084502a0 08064246 dd6a6908 dd60ec47

086de88f 080642b1 0874fdf0 08716ea3 08acf4e6 087596d9 08764b91 08737d6f

081a873f 08c35e2c 0875b4d8 dd60ec47 081a8710 08c4f6f7 0817a9d0 08172ec9

08445afd 086ffe7d 08702340 08445a48 08194340 0817d9bb 086d028b 086e66df

080642b1 086c966c 08194b47 dd60ec47 081a8683 0818a3b1 085d15e8 08052180

087145c9 08acf4e6 087b518e 0845142c 086ffe5e 08714afe 08194bca dd60ec47

08c37451 081a8912 08716856 086c87e8 08a630f2 084502d0 08064277 084401b3

080521aa 0805aa94 093416ae 08c3695a dd60ec47 08715bc0 0804ddd1 0818ea9e

08c50015 08717bf9 087b4ea0 08717bd8 080642b1 08172e03 08445ab0 087658dd

081a87c1 080643a0 08702589 0804cc82 081ab900 08445b6e dd60ec47 0874fea1

081a8683 08445b5b 0844dffb 0818ea9e dd60ec47 086e66eb 081a8732 08064353

081a8790 08442b96 08718afd 086c5c8f 0804eabd 0844dd5e 08719ea0 0843f6c0

08716e78 080eb180 0804ead3 08716794 081a8687 0817d9bb dd60ec47 086e00c0

08445a65 086e4728 0804eb18 08717d9a 0843f6c0 08442ad3 08172dff 0843f6c0

086ffe1b 0817d954 085d165f 0817d954 0844dc50 081a86d7 081ab1cc 0843f6e9

08445a59 08172eb9 08064af2 0875991e 081aa464 0875b4d0 081a88fb 080521c3

08170f0e 08717d88 0843fb9e 08172ec9 086ca887 0933b436 085d1580 0805220a

0819daf8 08a5eda9 085d2164 0843f6c6 086e0a50 08052184 0804cc74 081a8925

081aa532 080642aa 080644bc 08064d68 dd60ec47 08716571 087180f9 086c87c4

0875991e 08172eae 0871be0c 080520b2 08064246 0817fcb1 08172ec9 08064cb9

08172eae 0817fef2 081a872e 081aa476 086b8b28 085d159e 080642b7 0874d2f9

08170d89 08445a30 0817d954 08718adf 087b518e 081a872e 081a8683 08767c9c

086c50a5 08172ec9 08064a7c 0805aac1 0871bf7a 086c9bf1 08172dff 0805aa94

0817d900 081a872e 0817fe30 08194af6 0874d2ed 08172eae 0844dffb 08440150

081a87cc 0818a3a4 080522a6 08442c8a 08715db3 08172eae 081a87d0 086ffeaf

dd6a918e 08c4ff24 0817d9bb 08172ec9 dd60ec47 081a874d 0871818f 08440167

dd60ec47 08c35395 08a5edfc 086ffee5 dd60ec47 08c4f8f8 08715bc0 081945a0

086ffe5e 08172ec9 08764542 08c4f653 084420ce 0845141b 08715bc0 08055aac

087682a0 0936474c 086c87e8 087000e0 087607b9 dd60ec47 081a872e 087166c0

08c34d80 08172eae 087cac43 0818a380 08a6bb75 dd6a6976 08c35149 08183ef9

08172ea1 08c373ec 08064246 087025c0 08172ec9 08172dff 08717bd5 08c35d13

08c384e2 08445a3f 08c4f602 08445a65 08c4f69e 08c3743f 0843f6cf 08c35d7e

0806422f 081a8683 0874e825 dd60ec47 0819460e 08445a3f 081a87fd 087145ec

0933b870 dd60ec47 081944a0 081735f8 0817d954 0843f940 081a8683 0817fe31

0804cc74 081a8687 081a8904 0805aa94 08441e1e 086e0ead 086d22e4 08442bf9

087021dc 0871c48e 081a86d7 0933ba2b 080eb1bd 086debb3 080600c7 0876bced

0816ffac 08c4f6e7 08064246 086c1080 08172eae 08445a30 08717bf9 0874fa50

086debb3 084401a4 0843f6df 086ffe2a 08055a97 0804ddd1 0817fea3 08717bf9

08183e50 08172e03 0817d961 087167b4 08716e8d 08c4f729 dd60ec47 08715bc9

0933b436 0817d9bb 086d1795 086c635f 08183ef9 087171e6 086dfb80 086ffe5e

080643d7 081ab900 08c39f10 0818a381 0805aa94 0817d954 0871aa7a 08445ad2

08718afd 085d159e 0816ffbf 08445ad7 081aa698 086def80 080644fb 08737d6f

086c427c 0817d954 08194b81 0804cc8f 081a872a 081a8687 0816ffbf 08a6bc70

08717bd8 080600a6 084401ba 0806444d 08064246 087644f3 0843f6d2 08a630e0

0843f97e 0843f6e3 086c3057 081a8683 081aa532 0843f6e3 087bac00 08064246

0933b9ff 084401b0 08170dc3 0871664c 0874e7b0 0844dffb 081a88ea 081ab900

0817fe81 0817d91f 08a5edd1 08170fa2 0843fbac 0822b220 08445a3f 08717bf9

0844dffb 080642b1 08716e8d 08172dff 0817d954 08441db0 081a86d7 08c3b0b0

086c9066 0890b55a 08715bc0 086d21ef dd60ec47 08717bd5 08759519 08174a3f

0818a380 0843f6e9 081a86d3 08715bce 0843f6e9 08716e78 081a87b9 085d1ec4

0806010c 09347376 081ab1c5 08064af5 0871bbf0 080642b1 081a87f0 0875bf54

084401a2 0816ffbf 08442ad3 08064246 08717bd8 08450281 dd60ec47 081a872e

0843f6e3 08714afe 08172e07 0933b98e 087171a7 0805aab1 0843f6e9 08716e93

08064444 08c34f41 08717348 08064ade 08c4f703 0871c4ab 0876616b 081aa464

08172eae 081a86d7 08a5edfc 081aa558 085d15e8 087166a4 08064d60 0843f9a8

08c35e8e 081aa479 0845141b 0933b961 0804eabd 08c3a741 08172ec9 0805225e

084401ba 085d159e 08c4f8f8 08194bca 081a86d3 08717413 08172dff 08194bc7

08194355 086c90c8 086e68d9 08440196 086d2263 080600bd 0843f6c0 080520b2

084419e5 0843f6c6 0844dffb 08064246 081a872e 086c4a44 08717bd5 0844e507

08172f04 0822b244 08440174 08768f80 0818ea9e 08717bd5 0933b8b0 08064246

093472f1 086e49f0 08c4f600 081aa563 08445a53 086debb3 081a8935 086c9bf1

0933b961 081a88e3 08767ca2 dd6a67c8 0844dffb 087594f1 0844dffb 087047fe

0844018a 081aa460 0805f5cd 0875cf00 0843f6e3 08172eae 081a872a 0871bba0

08a6bb7e 081ab925 086c9bf1 086ffe87 081a86cb 0805aa94 086ca870 080643d7

08065bf2 0843f6eb 08716fff 080522a2 081a87ad 08715d84 087594f1 0843fb91

08c4fa2b 08c3b000 0804ddd9 086ffe5e dd6a91c1 08183ec5 081aa48f 086e66eb

0817fe81 08759165 08715d6e 08441a98 081a86cb 08064252 08c4f67f 084419e5

dd6a92aa 0843f6c0 08194cc5 080642b1 08172ec9 08c39376 0817d954 086de857

08c362b2 081aa460 086deeff 0843f6c0 086c36c4 086ca870 086c2c38 dd60ec47

0817d94a 08c4f67f 08718183 08445a31 08064246 0844e4f1 087021dc 0804eabd

081ab90c 081ab900 0817fef2 08c4f729 0871892f 086e68d1 08a5eda9 081a8687

08716ea3 08440156 08445b5b 0805aa9a 0844e507 dd60ec47 08717d88 08065d29

08445a36 081a88ea 08c4f602 08c4f67f 081ab1cc 0844dffb 081aa532 08174a3f

08445a48 08445a30 dd6a92aa 081a86ba 08c38050 08172eae 086c87c0 08c375bb

080642b1 081a872e 08172eae 081ab900 086ccc5e 08441970 081ab900 0843f6e9

086e6688 08714bca 0844dffb 0933b8a6 084401b0 0843f6da 0876051a dd6aa7e0

08c4f6e7 0843f6c6 dd60ec47 08445b65 08064a6a 081a86d7 08445b63 0804eabd

08065c14 0817fe4c 086f8830 08a5edf1 081a8969 08171000 0870259c 08065d20

dd60ec47 08715bc9 081ab196 085d1f02 0871bf77 08194a80 084420fe 0805aac5

0844dffb 08c4f68f 0844dffb 086e1108 0817d9bb 086d21b9 081a872e 0844017a

08183ef3 080642e4 08442bf9 086c87e8 0843f6eb 08c4f9f0 08445a30 08c3984d

08718943 0843f6e9 0933b9c0 08717e80 085d159e 086ffe5e 081a86cb 081ab900

081aa460 080642af 08194793 0843f97e 08064246 0844dffb 086c9bf5 081a8687

0804cc8f 08445a30 081ab196 08716f67 0843f82c dd60ec47 08760a7b 087166c0

0843f82a 08445b65 08170f11 08714bd5 086c9073 080643d7 086dffc1 0844dffb

086c6662 081a872a 081a86d0 081892f7 086d21ab 08717bd8 086de88f 0876616b

086c9bda 0817aa39 08445a30 08767b30 0844213d 087145b4 086c4280 08717bd8

0817d9bb 09347358 084502d0 08172eae dd60ec47 08764770 08064444 0933b909

0843f6d2 08716ea3 0843f6c0 0817d9bb 08760e16 0804cc80 08c35c80 087597e5

080600d3 08445a5f dd60ec47 080eb180 0817fef2 08767ee5 0890b560 08717bf9

087166a7 0817fe56 081a872e 08c4f6e7 081ab91a 081a87ce 0844dffb 081a872e

dd6a91c1 08717bf9 08700003 0875bcc1 087145e8 081a8687 0844018a 081aa499

086de846 080523d6 08172eae 0804ddd9 08060096 086c1080 087682a0 0875b4a5

08760a7b dd60ec47 0843f6c0 0817fe81 081a87b6 08172eae 0818edb1 08172ec9

086de84e 08172ec9 08716e8d 08441d83 081aa482 081ab925 08052038 0817aa39

dd60ec47 0817314b 081ab926 08715bc9 0844017a 08716571 080521e2 0844dd9f

08441e0d 08c35c37 08702340 087b8a10 08442ad3 081aa465 08172ec9 0933ba60

081aa532 08c501c0 0844dffb 0843f6c0 0819477f 08718368 08194387 08700560

0817d943 08715daf 086c87e8 0805208e 081a8687 0804cc64 08445b65 08172e09

087166d3 08715b69 0871c48e 081aa466 087002a3 081a88fb 0871aa13 0876929f

0844dffb 08716eac 08c4f688 086c94b1 083ea53c 08702346 08c4f653 08718896

08172ec9 0805208e 0890b551 081aa49c 08717bf9 08c4f67f 081aa558 08442ad3

086e6780 087598fe 081945d7 08442b5e 081ab196 083ec130 086debb3 08445ab0

08775831 0804d6ac 085d159e 0844dffb 08172ec9 08714acf 08c39190 08445ac9

08716f5c 081aa56e 086c968a 087171c6 081a8757 08194793 0806421b 086e67c9

0817fcb1 08445a5f 085d2500 0843f6e3 087180e6 08717bd8 081a8757 08c4f688

081a86d3 08758cc1 dd60ec47 0933b99b 08440196 0817d9bb 09346c0f 081a8a7b

08c4f67f 0875bd8c 08c4f8ec 086d1d86 08445a70 08172ec9 087035e0 0843f82f

08064ab2 0843f97e 0871485b 085d21b4 08445a30 0805aa94 08172ec9 081a872e

0843f82e 081a868f 086d1d86 08172ec9 087594f1 08172eae 086c968a 0806455e

08718890 081ab900 08a6bb7e 085d159e 08172ec9 080520b2 0843f6d2 08172e03

081aa460 086e0a62 081a87f0 080607e3 0806442a 086e08c9 08172ec9 0844dffb

0805fce1 dd60ec47 08c34e37 086debb3 08172ec9 0817d947 dd60ec47 08172e86

086d217e 0875b90a 08171000 09347301 08445a30 08702568 08767b30 081a8732

0806478b 0890b560 0844dffb 08172e9e 080522f1 08172eae 08c363bf dd60ec47

0875cf00 080600b0 081aa532 08716e8d 08060091 0933b982 0871be12 0875bd8c

086ffe64 08445b6e 0843f6dc 08194bca 0843f6e3 09346532 080642b7 080642af

0843f6fe 081a8687 0875ff4e 08194bca 0843f6dc 080523b2 0805f5c7 080641ec

08a6bb7e 084502d0 0844dffb 081a8683 08172ec9 08172eae 08c34b16 08065be0

081aa690 081a8683 081a879f 0817fe81 08c35e80 08445ab0 086c9066 08065be0

0875c4b9 0817d947 081a872e 08715b69 0804c214 0817fef2 0844dee1 087b50cb

0871bbd7 dd60ec47 080642b1 0817d954 087180e6 0844e511 087167c3 08195289

0873c07e 08753ef0 08c4f6db 086c9b9c 08194340 0818ea9e 081ab19b 0875991e

0873c020 08183e50 08170d95 081a8687 081a86bf 0871734e 08052236 0843f6c0

0804d670 08445a30 0875c27c 0818edec 0805240c 08172e9a 081ab900 081a8730

086e6525 08716e30 084420fe 08442bf9 08170ffa 0844dffb 086debb3 08172e03

08715d8f 08718113 0844dffb 087145b2 0844dc50 0844215f 0843f6e3 08442bf9

0871bf77 0871be78 08717bf9 08717415 08c36278 08194bca 08445ad7 08051fce

081a87b9 dd6a918e 08702382 0817d900 081951a2 0844023b 086e66eb 08194387

dd60ec47 09348571 087b8969 087145b4 081a8922 0933b961 08c4f8f0 08172dff

0844dffb 081aa545 0805aa94 08716571 086e6510 084502d5 080642e4 081aa55a

dd6a91c1 08172eae 080522f0 0876051a 081a8772 087b5041 08717bd8 081aa460

08737dd1 08c35d13 081a872e 086e135d 0804cc8f 0805f5bb 08716860 08172e9e

086ccb45 086c6394 dd60ec47 0818a3a1 087167ae 087591d2 086deee1 0806460b

0843f6d2 08445a36 086c9bf1 08716f8a 08445ac1 08442123 08c4f67f 0804ddd0

080eb1bd 081a8901 08c4f79a dd60ec47 08722c57 08174249 08172ec9 0805220a

08172ea1 08442ad3 0804cc8f 08c4fa2b 08170dc0 0817fea3 08065d29 086c9bf3

08442ad3 0818ea9e 0875bb9e 08716694 086d177e 081a8922 dd6aa66f 087b502e

08445a31 08c34a72 081a8904 0806428c 08715bc9 081a892e 0936477a 0843f6d2

081ab189 08183ef9 0817d9bb 0844dffb 0843f6e3 08716f67 0818a3a4 0817fe31

086ccc5e 081ab90f 0843f6e3 086f34f0 0844dffb 08758cc1 092e9330 0805220a

08195289 0843f936 086c2c38 08a6bbc0 08716f67 08445b79 086cc321 08172ec9

086f337f dd60ec47 0805aa9a 081a87b6 0817fcb1 081a87fd 08717e42 0817fe56

08715d79 0804cc8f 087b5259 08716e78 0843f6dc 08064991 09346c12 08717323

087b8969 08442af1 081749e1 08445afd 0817fef2 081aace0 0875d590 080522f0

087167bb 0817fcb1 0844df39 08064a7e 08442181 086c87e8 dd60ec47 081aa465

08c4f67f 08172ec9 080643da 081a86d7 086c36c9 08442275 081aa558 086ffee5

086d2263 08445a45 0804d670 08445b67 086c9bf1 087b518e 08c362a0 08716e30

08064d77 081aa532 dd60ec47 081a871d 08171000 085d1668 08716eac 08759102

086c41e9 081ab1c9 080eb18f 0933b925 0871bcdc dd60ec47 086c9076 dd60ec47

08c35c37 081a86cb 08715d37 08718ef6 08a5edf0 08717f14 dd60ec47 0876057b

0843f6dc 0843f6e3 08170f25 08064a9d dd6a918e 081a8683 086d2239 dd60ec47

086debb3 0871485b 081a872a 0843f97e 080642a5 08716680 08c36430 08445a31

0818a380 0806460b 080644fb 086c87ce 081aace4 08c3928d 0873c020 0871669a

0817fe81 08445a30 0817d954 081a8687 dd60ec47 081a86d7 dd60ec47 08718eec

081ab90c 08704a60 086c87e8 08170de5 08440174 08064d86 08170f01 08c398fd

0875b380 0817d954 081a872e 08714ec1 0933b9c0 dd60ec47 08442108 086c36c4

dd60ec47 08183ef3 dd6a91c1 08442d9a 08c39f0d 0875c3a1 08172eae 08441e2a

080642aa 08717323 08c36021 0843f6e3 0933b97c dd60ec47 080521ff 08194cde

086c8a41 0933b8d6 dd60ec47 086f337c 08064ab8 08064a81 0819dae0 081a872a

0843f6f8 08170efd 084419e5 093472f1 08c35aff 086ccc5e 081a873f 0817d954

dd6a6902 087b5180 08715b44 0843f6e9 085d2100 08716ea3 08170dff 083fcffc

086e6511 08445a67 0873bf10 08172eb6 080521aa 0843f6da 08c4f688 08064af5

08c4f65e dd6a91c1 086c5acb 08442174 08717e34 08716f3f 0843f6d5 08445ab0

081a8922 08759232 080642b1 081a892e 08a6bb7e 08229d20 0844e4fd 0805220a

0873c020 0843f6c0 0804cc92 08445a31 0817a9d0 08c4f8d8 08c3b294 0933b958

08172e9e 0874d11d 08718b90 08174a3f 080521e2 08172eae 08c35a40 086c968a

08714865 0843f6e9 08172eae 08c35c37 0843f6e3 0844dc50 087b8a10 0844dffb

081a87f0 0817a9d0 081a872e 0817d954 08c3744e 08c34a29 081a88ec 087023aa

081a872a 081a872a 08c358cf 08064c6b 0805f5bb 087025a7 086e67b8 086c9080

08055d9e 08767682 0805f5bb 0871bca7 080520fb 08172eae 08172eae 08445aff

086e137d 0816ff50 0817d9a0 0875991e 086fffff 08a5ee10 dd60ec47 0875be8d

09343af0 08c4f653 0871c710 086de857 081a8687 0817d9bb 087b518e 08715bc0

08442133 085d1581 0933b99d 080600e2 08753ef0 08a6bbc0 08445b66 08064a36

08189302 086f3403 08c4ffa7 0875b9f0 08715d8f 085d2164 086c3590 081aa545

08716f67 086e66d2 0843f6c0 08c362d2 085d2164 08064c64 08c4f624 08194652

08060090 086c9bc9 08065be0 0816ffb3 081aa698 081a892e 081aa860 086ffe54

08445b6e 08c4fefe 08065be0 dd60ec47 08055a97 086c8601 08442272 081a872e

0844dffb 08a6bb7e dd60ec47 08c4ff66 081a8732 08717f04 0843f6dc 08172eae

dd60ec47 086cdbd6 08c35df1 dd60ec47 dd60ec47 08c39284 086c6664 08700ea4

086d17bf 08737d6f 086c5c4b 081a8912 093473a0 08194bca 08194bca 08170fe1

087025a7 0818eaab 08716e8d 08c3a013 08700603 08716780 08445a53 0843fbac

0844dffb 081ab411 08194387 08c391ff 0843f984 084401a9 087b518e 0806434c

0818ea9b 08717bd5 085d216d 080eb18c 086e0775 08717bd8 0871725c 08442141

08717bd5 08172ec9 08c4f602 08442ad3 0871bf83 08714778 08441971 087cac00

08445a36 086ffe4d 086f8870 08064252 0843f6dc 0843f6c0 086c64ff 0805220a

0870235c 0844e511 08716872 08172eae dd60ec47 081aa690 08715dab 086e01fd

0843f6c0 08716e90 086debb3 08194773 0933b9d6 08c35705 08716f67 08716e96

08445b0d 08052188 08055a97 08c4f9f6 081ab903 08c4f6a0 0817fce8 08194652

08703600 0818a3a4 086c9be4 0816ff51 087167ef 080642b1 087000ef 08172e9e

080521aa 084401a9 0805aa94 dd60ec47 0805aa94 081aa464 087b5259 0806460b

dd60ec47 0871818d 0875cf00 080643a3 0804d6ac 08170d95 dd60ec47 08193c00

08737d6f 08194cc5 081a8683 08172e9e 0844dffb 0933b43f 08c39743 081a891d

081a872e 08c34e01 08718ee4 08714bfd 08064246 0871bbf6 0817fe4c 080642aa

0871c723 09364757 08716860 0874e7e7 dd60ec47 086e67b8 087510f0 087185a5

0875fe9e dd60ec47 0804cc8f 08172e9e 0843f6eb 08769ca8 08c39d76 0844dffb

081a894a 08445b67 08717ed9 0817d9bb 085d15ea 080521db 08c35c37 085d15e8

08acf4ec 08c39d8f 0870023f 087166c0 08759765 09341093 086d1180 08715bc9

08716e78 0933b97c 081a86d7 080643da 0844e511 dd60ec47 0845141b dd60ec47

086c3570 0936474c 08172eae 081a87b6 08759232 085d1f30 08172ec9 08445a70

08445ac9 09346b20 086ffe5e 0871656e 0875fe44 080520b2 08055a9c 086ffe02

0805aa97 dd6a92dc 0805aa94 08064af5 0876bc81 dd60ec47 087185cc 087172ce

0845141b 081a872a 08718890 08702550 081a891d 0844017a 08065d21 081aa532

081aa476 08172ec5 085d2184 08714771 086ffe5e 08c5010e 08753fff 08715bb6

dd60ec47 0804d6ac 0843f6c0 08715bc5 08065be0 0817d9bb 08737de3 0844dffb

080643a2 086ffe5e 0805aa94 0804ddd9 08714851 0805231a 08172ec9 080643da

08174a5d 086e65c5 08c375bb 08715b44 086deb90 0817d930 08714771 086ffe5e

0871a8c0 08170d90 08c35f04 08052188 08445a48 08714830 0817d901 08445ab0

dd60ec47 dd60ec47 08064246 08170ffc 08c4f67f 08065be0 0843f6c0 081a8730

080642af 08064a9b 0843f95b 081ab90c 0871c4a0 086ffe5e 0818eaab 08714580

0844dffb 081a8901 0933b870 08717bd5 081aa545 087c0aae 087682a0 081ab913

086d21ef dd60ec47 0804cc8f 08c3984d 086c36c4 086ca28a 08c35e20 0843f6fe

080642e4 087cac00 087150d0 08183e73 08172ec9 0817fce8 0805aa94 0844dffb

08194355 0875cf7e 08195289 0933b431 0817fe31 dd6a918e 081a8683 086c8802

0890b560 08172ec9 08c363fd 081943d2 081a88f4 0844dffb 08a5edd0 0817d9bb

0805aa94 081a872a 08c4f6d8 08442b61 0871bcbf 08064246 087b8969 081735f8

dd60ec47 08715b69 080600b8 08172e82 08194bc7 085d1668 08172ec9 08750124

086ca870 081ab926 080643e6 08172eae 081ab90c 081a8a50 081ab901 081aa499

085d15e8 085d2161 081a8935 08445a67 08716e8d 08737dde 086df389 086deea0

08172ec9 dd60ec47 08c35e59 080522f0 08a6bb8a 08715d71 08442bf9 0873bed8

08c4fa9b 08716f67 086d627d 081a87fd 080642b1 08440185 08c4f67f 0818386b

08716e42 081713ed 08716833 080641ec 08064246 08442ad3 086d2105 081aa698

086de88f dd60ec47 08760e16 08445b0d 08442bf9 086ffe5e dd6a697e 0843fba1

08768d4a 087188cf 08064246 0817fe31 0843f6c0 087b8951 08767c0a 0817d9bb

086e67bd 086d21ae 08445a30 08c4f67f 08172ec9 080642b1 08172eb0 081749e1

086c36c9 08445a31 087510f3 086df40b 0805aa94 0843f6e3 0817d906 08716872

086c88d4 086deeee 081a872e 081a86d3 086e68d1 080642e4 dd6a918e 0822b244

0874d2ed dd60ec47 08702355 08c500bb 08c5000c 08442ad3 08715d37 081ab196

08700273 08717f3b 08442ad3 08172ec9 08704d11 08717bdb 09348571 081a8793

086d28d1 0806444d 080642e4 08441970 0817fe58 0804cc7a 08759404 086c9076

0875b932 081ab1cc 08441a98 081aa558 08172e9e 0874fa50 081ab926 08c372b0

08c36880 080600d3 081a8687 081a86d7 086ffdf1 08c4f67f 0933b8ad 08767630

08440150 081ab901 083ec13f 081a8683 08445b67 087185a1 08052075 08445a65

081a872e 081aa690 081a872e 0817fe81 08450281 0843f6c0 08715bc0 dd60ec47

086c8596 081a86cb 0844dffb dd60ec47 086c5bf0 086f8880 080600d3 087c0aae

0817d934 081749e1 08052180 08715d48 0805aa94 086c6654 0871bddd dd60ec47

08716620 081ab91a dd6a917c 081aa479 086e0ead 086c94b1 081ab19b 0933b96d

08716732 0933b9af 081aa460 08ac155c 081aa492 08c4f710 08064507 085d2186

086c88d0 0843f6d5 08441db0 0843f6eb 08c392c0 08716e8d 0817fe81 081ab900

08c36572 084419e5 08195289 0843f6e9 086c90c5 0817d9ad 0843f6d2 0805aa94

0844dffb 08445b5b 087b5020 08194387 08052180 08172ec9 081a895d 081aa482

08060091 0806451e 08172e9e 0804ddd0 08c35dd3 086d21ab 0817d954 087166f9

0871bba0 087597e5 0890b562 0843f6e9 08064246 081aa558 0844dffb 08c4f653

08064353 086d02d6 081a87c9 081a8901 081ab900 087682a0 08055aa2 081a872e

085d2184 081aa53e 0933b883 081ab91a 08445b65 08c5000c 087026e9 086ffe5e

081a86d7 08715b81 0806427d 08737d6f 0843f6e3 085d1641 08064246 08445a31

086e4710 08172ec9 08445a31 08c35d05 0870234f 08716550 08064af9 084401bd

081a86d3 08194c81 093472f0 081ab1a5 081a88e0 0843f6e9 0806423c 0805aace

08173630 086c64ff 08759753 dd60ec47 0871818b 08445a30 086c8a94 08194b74

080522f0 081a8901 081a87c5 081a86cb 08716571 08718bca 086e4710 087594f1

08172ec9 081a87ab 08717bdb 083ec197 08c34f52 0817fcb1 08718650 08445b67

081ab90c 08170de5 08714afe 085d159e 0818ea9e 081a87cc 08c3b235 0817d9bb

085d15e8 087b5020 0805f5bb 08060096 081aa532 08064286 08715bce 0890b560

081ab19b 08445b67 0818ea9e 081a868d 084419e5 08052223 081a8689 08715d45

081a8714 0817fe81 08445af6 0805aab1 086d6ec0 dd60ec47 08172eae 08c4f602

086c5d26 0871be4d 085d2184 08716f6b 08a5edd0 08767630 086e66c3 081ab900

dd6a7740 0844dffb 0871a950 08716eb2 08440150 087b5200 092d9f52 08195289

085d2184 08194387 0806010c 081a8a9f 08440196 081a872e 08c4f7fb 0843f6f0

0844dffb 087167c8 08445a30 081a872e 087babd0 081ab900 08715bc0 08064af2

081a8726 086c4650 0844e511 0871bcde 08194378 08c392c0 08c4ffa7 081a87b6

08442187 08064564 0844dffb dd60ec47 087598f0 081a871d 081a86d7 086c421b

dd60ec47 081a894a 085d159e 08440259 08442d9a 08c356ff dd60ec47 08055a80

0844dffb 085d1668 081a8683 08442b0d 08064246 0816ffbf 08703601 08445a53

081a8942 081a873f 0876bc51 081a872a 08055a97 08716ea3 08172eae dd60ec47

0843f6f2 086ffe5e 0844dffb 0843f6e3 08c372b0 081a872e 0844024a dd60ec47

08700102 08445a3f 0875abc0 081ab900 086e67b8 087167a7 086e65e1 08717bd8

085d159e 081aa860 08c36430 081a86cb 08445b71 dd60ec47 08c3b001 086ccb54

081aa461 08c36262 08a63122 080522f0 09342b25 dd60ec47 dd60ec47 08c350e0

086c9bf1 081a8687 0843f6d5 08064cb9 08064246 dd6a91c1 0804cc91 08064af2

081a86cb 08170efd 08c4f653 08c350e0 dd60ec47 08c4f698 08055a80 08c39221

081a87c1 08716e88 08445b0d 08715bc3 0843f97e 08c36f22 081a8687 08717e35

0817fef2 085d2161 0936474c 081951a2 08c35c80 09340ffc 0844e511 dd60ec47

087c0aae 08c39f0d 0843fba1 0844dffb 086c36c4 0875cffe 08064359 08716f67

08716f67 0805aa94 085d15e8 085d1581 dd6a6726 080642d5 08716f67 08716f67

085d1f0f 0843f6d2 08c3625b 086e0cab 08441975 08715d79 08c4f60b 080600b0

086e0a50 08c363f2 086debb3 080523e1 080520fd 0843fbd4 08c35e59 08441db5

08195289 086e65d8 08c4f73c 0817d9bb 081aa570 08c4f6a0 086c9080 0875b5bf

0817d9bb 0875e2b1 0844dffb 081ab1bf 0805213f 0933b961 0933b948 08194bca

08172eae 08717dda 086c87e8 08acf40a 080642b1 0890b560 08c36021 08064555

dd60ec47 0875bd7d 0805aad0 dd60ec47 08c4f688 dd60ec47 08c35d7e 0871bddd

081ab900 08c36021 08442ad3 084502d0 081a87fd 08064499 081aa532 081aa49c

081a892e 081ab90f 08172e03 085d24fc 08445a3f 0875b6ed dd60ec47 080643d7

080642b4 08c4f67f 080521aa 0843f97e 081a872e 086c9bf1 0843f97e 08051fa0

0933b955 0804d6ac 08c4f710 08170d89 0818a380 08064550 08445ad7 0843f9ab

08c4f6a0 08440196 081a892e 0871c231 08060091 081aa558 08c4f67f 08716621

08716721 081aa482 0933b9d6 086c9066 0871a991 0805f5c1 08c35d51 08170d90

0843f6c0 085d24fc dd60ec47 0818ed90 080521aa 08173130 08194cc5 0805aa94

081a8714 0890b55a 081a86cb 08715b69 08717ee2 08170de5 0843f6e3 081a87ce

08441a98 08172eae 086fff40 081a8687 086ffe5e 08716732 0873c020 087002ea

086c88d0 086c87e8 080643da dd60ec47 081aa465 086e66df 08172ec9 0805aa94

08717348 dd60ec47 081a872e 0805aa94 08c372b0 086c3057 08c35f92 0933b430

dd60ec47 08445b63 09348550 086c36c4 08051fa0 0817d954 08715b81 086c94c2

0806009f 08445a30 085d2164 08445afd 08064a9d 0806428c 0871896d 08717bd8

0818ea9e 086ffe54 08717bd5 08064d8d 08064c6b 080521bc dd60ec47 086c90c2

dd6aa66f 0844197f 08715b81 08c4f6bb 0843f6e3 08064221 0933b8b0 086debb3

08c39271 08717bfb 086e49f0 08064246 08195289 084401b7 08052036 0843f6c0

086e68d4 08c35aae 086e65af 08442ad3 08c4f67f 086ffe5e 08172eae 0805220a

dd60ec47 086c2240 dd60ec47 08764450 0805aac1 081aa54c 08060090 081aa466

0805220a dd6a91c1 08445a30 08c4faa7 0933b43c 08c363fd 0816ff68 08195289

0874fe80 0805220a 08716e8d 0875c440 08172e9e 08718366 081a868f 086ffe5e

0843f83a 0816ffb3 08172ec9 085d2184 08c34b40 08716564 08065c14 0843f6c0

0843f6eb 08717323 08064550 086c2c38 08764b91 0805aa94 08172df6 086c5813

08064a31 0806430f 0843f6e9 081a8938 08715d37 08715b44 086c6662 0817aa35

0843f6e3 08c39226 0875bd6c 08716541 08717bf9 08064302 081a87b6 dd60ec47

08442ad3 081a8710 dd60ec47 08442d9a 081aa46f 0871c72f 0817d9bb 08c4f6a0

0874d2ed 08172ec9 dd60ec47 086c3380 086c9076 0843f6c1 08c381f0 0805aa94

0843f834 08172e86 086c9066 0875967c 08716540 08716e8d 08718afd 0806424c

0843f6e3 dd6a92c1 08442106 08715db3 087c7aa0 0933b430 086de8a0 08445a53

081a892e 0843f6d2 0875ba00 081a8683 08441e36 0844dff3 0844dffb 086c967f

0818ea9e 081a872e 086c1180 dd60ec47 08445b58 08172e86 08064221 08c35aff

0871bba1 08c398fd 0871818d 08174a9c 081ab90f 081aa57f 086c9be4 087167c3

dd60ec47 08445a36 08716747 0875dd00 080642b1 0875befe 080523a8 08c367ab

0817d954 08440174 084401bd 087b8969 0817d954 08c2240d 086d1f60 0804d670

08172ecb 081a8687 081a86d3 08170efa 0817fe30 08442155 087cac3a 0871889b

08c50167 086ca28a 08c36618 dd60ec47 08442ab1 086ffe87 081aa534 086ff960

08194bca 08714774 0875a5ff 086c3683 081a872e 08c38a14 08c36569 081aa532

086c5d26 085d15e8 0817d900 0843f6e3 086df389 08445a30 0933b96d 081ab91a

0843f6e3 086f8830 0873c047 086c35a0 0871aca1 081a8683 0817d984 080642b1

08c3b190 08c4f602 08055a97 dd60ec47 0817d9bb 081a8980 08172eae 086c88d0

080600b0 080522f4 086ffe5e 08442bf9 0843f6e9 08064550 08714e60 08715bc5

08c35ef7 087167ef 08170da9 086c92c0 087166d7 081a871b 085d15e8 08445a30

0933b998 08172ec9 0933b97c 08170d70 08715daf 08055a97 08716f67 081aa53e

dd60ec47 08445a48 08170de5 08c36235 0843f6e3 081a86d3 08753ef0 081a87b6

09348571 08c4f68f 080eb19d 08717e35 0817fef2 081a87c1 0844dffb 08c4f7a2

0843f6eb 0843fb91 0843f827 08714580 081aa57f 08717bd8 08064366 0805aaa0

0844dffb 081a87d6 0816ffbf 08716541 0817fe81 081952a1 0843f97e 086c4ffe

0843f83a 0871810d 08c3538b 086c88d0 08c36262 08183f02 dd60ec47 081ab90f

08172eae 0805ac00 0843f6f2 081aa698 08172dff 0843f6c1 08715d4d 086e68d1

086debb3 0843f6fe 086c88d0 08c3514d 08064221 dd60ec47 08716ffc dd60ec47

08c4f6e7 08178046 08172eae 0843fbac 08064507 086df389 0843fbac 0817fef2

08a5edd6 0817fea3 08715d4d 08064246 0875bcc1 dd6a9191 08714afe 086c6662

0844017a 0876bc10 080643d7 08064246 08172eae 081a890b 0875c183 081ab1c9

081aa545 0843f825 08716f67 08064274 0871735a 0806456e 086e0ead 080600c5

081951ba 08064aed 08c4f60b 087596ee 08737da3 08716f67 081a892e 0844dffb

087145b4 08c4f67f 0817d9bb 080600d3 0816ffbf 0870259c dd60ec47 086cdbec

086debb3 08acf4ec 08717bd8 086ffe7d 086ffe64 08c5009f 08715db3 0843f984

08065be0 080521aa 08c4f6a0 080eb180 087b8a10 08715b81 087171c6 08440174

08702140 08715d4d 084419e5 087150d0 081ab925 08442ad3 08702589 0804eab0

0890b560 0870238f 08170ff7 085d231b 080643da 081a872e 085d2500 08c4f6e7

093472f0 084401ba 08737d6f 08440199 08716f67 08172eae 086bf7a0 085d2164

0875d04f 08c34b40 080643d7 08700d43 08172dff 086fff53 08715d93 081aa466

08716ea3 081a892e 0818ea9e dd60ec47 08183e50 086c9bf1 dd60ec47 086c968a

08c4f6a0 08172eae 0844210a 0817fef2 08064550 084401d8 0843f6c0 086df389

08717bd8 08445ab0 08c4f653 081a8ac5 081aa58a 0804eabd 0933b961 08183e73

083ec130 08717f19 0844e511 08194bca 081aa532 0843f6c0 081944a0 dd60ec47

086e0fe3 0817d954 0843f6e3 08718f0b 081aa57f 0844dffb 086d1d86 0871bba0

0817d954 08acf420 086df2b9 08052131 08717333 08052307 081aa690 081a871b

08718afd 08714830 081aa532 0818ea9b 085d2184 080600b0 08702386 08064246

086dfb70 08172e9e 08c34d40 08714771 08051fac 08172eae 08716f67 08445af4

08451441 08172ec9 08445a4d dd6a68f4 0804eab0 08717326 08445b58 dd60ec47

08055a97 081946ae 08715b41 08715d48 081a8790 0890b560 086ffe5e 08189327

08194b53 08716f67 08717bd8 08c362ea 086deb96 0843f6e3 081a873f 08a6bbba

08064221 084420d1 083ea444 08c35b7e 08194357 08442c74 086c90c5 08442d9a

0805aab1 08442ad3 08171071 0804eafa 08194bca 086cdb66 0843f6d2 080eb1bd

085d2184 08052207 08715bc5 08c35ea2 08716e8d 081aa58a 081aa54f 08064435

08171071 081a873f 08718ef6 087652ad 081aa465 08718130 086c87e8 081a87ce

0874fe9d 0817fef2 0933b96d 08714b01 086c9bf1 0871889b 0806438f 08c4f653

086e0d90 0843f6e3 08064246 0871bba0 08445ad7 08715b69 086c89d0 08716f67

0844213d 08051b21 085d15e8 0805aa94 08051fa0 086c2240 080642b1 dd60ec47

08150663 0875403e 0844dffb 08c35f00 08c4f653 0818ea9e 081a8687 0936474c

0871bc03 081a88e7 0817d984 08c4fee6 08c4f87b 0874d2d0 081a87ad 0817fe30

08717bf9 08716ff0 081a88ea 08170efd dd60ec47 08052125 0822b2e6 0843f9d5

08764dc9 08717e80 0871bbd7 08442e20 086f3403 081a8683 08065be0 0936474c

08c4f685 08445ab3 0843f6e3 08c375bb 081a86cb 084401bd 0817d901 08717bd5

085d15e8 080642b1 08445b5b 087166c0 081aa563 08c37705 0873c020 08c4f653

0933b97c 086ffe87 0871ac7f 08445afd 0845142c 08718183 dd60ec47 084401b3

086e04d8 0817fe31 081a8721 08700d46 087594f1 08064520 08194355 0805241a

08183ef9 08715d82 0843f6e3 08445ab3 081a87c1 0806402c 0816ff79 dd60ec47

080600d3 08716f0d dd60ec47 0817fef2 084401a9 08c4f67f 08171086 081aace0

08715bc5 081a872e 08758c7f 086ffe12 081a868f 08445a30 0874d2d0 08716e8d

081a871d 08442b6e 08c36278 081710a2 dd60ec47 087021fe 081aa698 0871bf77

dd60ec47 0871c48e 08172eae 0817d9a0 08c3984d 081892e0 0871bf77 0804cc7d

0875ff51 086debb3 081ab926 086e0a50 087b51f7 0933b961 08195ef4 08717bf9

08442bf9 0843f6cf 086c6642 08715b7b 08c398fb 08c37442 0817107f 086c9066

09348571 086df389 08716e90 08717ed9 08445b05 0804eacb 081a892e 086c9bf5

0805aa94 081a8687 081aa532 086c87c0 081aa860 0817d9bb 085d2184 081a8687

0816ff79 080642b1 dd60ec47 086c6614 dd60ec47 0843f6d2 081a87ce 086df389

087b8969 0875f2af 08737d65 081a8710 08764da3 081a88fb 0875bcc1 081945a1

0843f6f2 08c4f67f 08715b69 081ab1bf 086debb3 081a894d 08170f01 0871687b

080521c3 0844dffb 08051fa0 08717bd8 081a8683 081ab1a5 081a872e 081a86d7

08445b6e 08172eae 081a87b6 086ffe26 08172ec9 0844e4f1 081a8790 08441975

08445a30 dd60ec47 08172ed5 08194afe 081a87c5 081a8687 0871ac7f 081a8953

08445b67 0844dffb 08172ec9 dd6a91c1 08c35d23 081a872a 0876952b 086ff9fe

086c66c4 0871aa7a 0843f6c0 0804cc7d 081a8689 0933b97c 080600b0 08c4ff03

086e68e6 0871bbbd 0804cc82 087145c9 08a67fa0 08442272 08194cc5 08064246

0805f5bb 0817fef2 0818ea9b 0845141b 0844dffb 08718ef6 08173174 081a8687

0806467f 0843f6d2 08445ab0 08a630e0 08172eae 08717ccc 08064cb9 0844e4f1

086e66eb 0871bddd 08c393da 0817d954 dd60ec47 086ffe87 08064af5 08c3744e

0844dffb 08172eae 0933b995 08445b58 0844017a 08445b0d 083fd001 0875b4c9

dd60ec47 080600b0 086c66b9 08c362ee 08c4f6d8 087167bf 08172eae 08716fff

0818edc1 081a8935 081ab1af 08758c7f dd60ec47 0804cc80 08c36404 081a8683

0817a9d0 086c9bf1 08759519 0843f827 0843f6e3 08172eae 081ab1c5 081aa466

0843f6df 08c4f8ec dd60ec47 0810e91c 0817fef2 093472f1 08760884 081a8770

085d159e 08717b4a 086c9bf1 08064af5 081ab925 dd60ec47 08445a31 08718b0c

080642b1 0818ea9e 0933ba01 081a86d7 080523be 0843f827 08445b7f 08717323

0844dffb 080521cd 08440156 0805aab1 0818a3a4 086c3590 08717ccc 080647b1

081a8942 086ffe5e 08440196 0843f6d2 08194b7c 081a86cb 08064246 086c9d70

0933b961 081a87b6 081a8790 0805aa94 081a8750 08c35aff 08445ac9 08c4f602

dd60ec47 085d219f 081943d2 08171086 081a868f 080642af 0875ff64 08716e8d

081a86cb 087167ac 0933b9d3 086d21ab 0871bc91 08717bf9 08c35dc5 086c5813

081aace4 086c87e8 086e66eb dd60ec47 08743bc1 086c46ed 080642b7 0844e4ef

0843fbd3 086c9bf1 085d2164 081a87b6 08445a30 0873c020 08170d82 dd60ec47

0843f6e3 0843f6c1 08171086 086e4710 08060091 0871bddd 0806485c 08194cc5

080642aa dd60ec47 081a873f 081aa532 dd60ec47 08716e9f 08714784 0804d670

086c5428 086de8db 08445b67 0844dffb 08c35708 08442bf9 08c4f65b 08064283

086de836 0874e800 0806434c 0817d900 dd6a91c1 08442b71 dd60ec47 080642e4

081a872e 084401a9 081a8904 08194bca 08700560 0875c3a1 0817fef2 081a86d0

084502d0 0818a3a4 08c4f6e7 085d2181 09347338 081ab926 081aace0 086e68d1

0875b4d8 0875cf98 087594f1 08442c74 08715bc0 08716e8d 086e66eb 080642e4

08183e92 086c968a 0843f6d5 08a5edf5 08723282 081892e1 086dec14 0804eac5

0817d90c 087047d9 dd60ec47 08716f64 0875be93 0818eaab 0817fe81 086ff960

08717bd8 08c35705 081aa48f 08716571 08764487 08764da6 08c4f9f6 0805220a

08442174 08172eae 0843f6c0 0875b9f0 08718916 081ab1b5 0817d9bb 081a87ce

08c3771a 080642b1 0844dffb 0844e001 081ab196 08171071 08172ec7 08716f67

08715b81 081aa698 dd60ec47 087025a7 081a8687 08052146 081a86d7 08445a53

081a8687 08717e45 08441970 086ffe5e 087605b1 087180d0 08442ad3 086c4280

08172dff 08171071 087682a0 08172e03 0843f6e3 0817d901 08717bd5 0817fcb1

08172df6 081a892e 08064af5 08c4f6a0 0817fe4c 0844217f 0817101d 08718132

08717bd8 08c35a41 081aa466 08194418 08715c6b 08716fe4 086d1f24 08064550

081ab196 085d2136 081aa465 08715d7f 08172eae 087185c0 0871bcc2 08065bff

08172ecb 086dfb20 08716680 08445af0 0804d670 08c4f6f1 08051fa0 08442b85

080600c5 086de841 08716e8d 08195265 081a87de 087594d0 086ff97e 08702596

08441e2d 08065be0 0806456c 08a6bbae 0805f5bb 08716f67 081aa86a 0843f6df

0806425b 081a8942 081a86d3 086c669c 0817fc60 086e66ab 08064cb9 093472f5

08716f67 086ce381 0843f6e3 081aa465 087606a3 08445a36 08715bc0 0933b961

085d1eb0 087b522e 08440174 08183e80 081aa54f 08715d7f 0933b961 08170de5

0871bba0 08718366 086c36d8 0871bbf0 087180ef 08172e9e 0844dffb 086dfb20

08c4fa2e 08065bff 08174a3f dd6a6726 dd60ec47 0818ea9e 081a88f4 0871bea2

080641ec 08c3aa85 0871811a 08c4f67f 0805205a 08c3b0b4 09341093 085d2184

09347384 0844dffb 0817fd2e 086d178b 0871486f 08064020 0805aa94 081946bc

087501a0 08715d37 0806424c 0817d9aa 0844defe 08065c14 08715b81 08442b45

081aa558 08716f67 081a87de 08715b81 085d1f90 0818a3a4 0817d9bb 081749e1

086f8830 08064550 086d2885 08445ad7 08704a60 08052180 08064246 08737d6f

0805aab1 081ab900 dd60ec47 08442d9a 08718adf 08715bd7 085d2100 081aa56a

086d217c 081945d7 081aa499 08715d93 08170de5 081a891d 0871654d 0933b995

0844e013 0933b430 08c34d8c 0844df04 08c35abe 081a8925 0875b5e0 085d2161

0875dd7d 08c4f6f7 0817fef2 0817feea 081a872e 0805214d 0806439d 0806460b

0871bba0 0871aca5 0817d954 08060091 086f34f2 081ab900 dd60ec47 086ce381

08442b16 dd60ec47 0805aa80 081a872e 081a88ea 081aa558 0806423c 08c34b16

08a6310b 08194652 081a8687 081ab901 081a8730 0817fe81 08065d2f 087025a4

086e67fc 0875c123 086c667c 081ab901 08c398fb 08c4f6a0 08750124 08170ffe

0844dffb 08c4f8ec 0817d900 08051e5b 08717cfa 08442ad3 0843f6c1 0817fe30

0843f6e3 081aa558 08715d4d 087145c7 081ab900 0843f6d2 086c9bf1 08c34910

081aace4 08172eae 0805208e 086bf6f4 08c3602a 08064af5 08194cc5 dd60ec47

087171c6 08716ea3 0818ea9e 086dfb20 0875d590 08714771 086c41c0 086e67c9

081a8687 08717bd5 086e44b6 08c3627e 086c5c18 08c35e6f 08065be0 0844dffb

08c4f68f 08c3984d 081950ca 08759f60 0805aac1 086c41cd 08c3b03c 08065be0

08170dc9 081aa558 08c4f653 08172eae 081a8735 08064394 086debb3 086c9066

08c4f653 08445a59 081738d8 0817feb4 08055a8f 08055a80 08170ffe 0818a3a4

0874fdf0 081a872e 08445a4d 08717c20 086e0a50 09348551 08445ab3 08060107

08172ec9 085d2184 08c370b5 08c35d23 08717330 0873c021 dd6a9389 086deb90

dd60ec47 08170dfd 08700126 08c35c37 085d2186 0844021d 086e6688 08172ec9

081ab905 081aa54c 0817d9bb 08716f88 0817d9bb 08441e0d 093472f0 0875b4c9

08172f16 0875dd05 08c4f60b 08441a83 08c500b3 08c350f4 081a872e 081a872e

08445a65 081aace0 08714afe 08c4fefe 081ab900 08174280 dd60ec47 0817d9b3

086ffebb 08174a3f 08173637 08064020 0843f6e3 08172e03 0843f6e9 0805218e

085d1eb0 081a87de 0843f6e3 08909cfb 0843f6eb 086e08d5 081ab900 0871be51

0818edc1 08194bbc 081a8757 0843f6d2 081a87b6 08c3514d 0844dffb 0876027f

086deac0 080642aa 08445a30 0804cc92 08716564 08a5edf1 081a8925 08718130

0806486a 0844dffb 087000e0 0933b9b3 08714774 08716f67 08717bd5 08170dc3

08064246 0870239f 087180e6 08715b69 dd60ec40 08172eae 0817d954 0933b9fc

083fcff0 08445ad2 08172eae dd60ec47 08c4f7e7 081ab1cc 0817d9bb 0844dffb

08758c10 081ab926 081a868f 081a87fd 086c2c38 08194847 0805aa94 08c4f9ab

08c4f653 086d1180 08170de5 08445b71 0871bc00 0818ea9e 08172eae 0817d9bb

08703600 08716571 dd6a918e 086ffe87 08a67580 0936474c 086e67dd 08065d2f

087b5180 08172eae 08715d4d 08718afd 081aa532 08c4f6e7 08c35f92 081a86bf

080642b1 0817fe81 0805220a 0933b955 08172ec9 08c4f6f7 085d159e 08716856

dd60ec47 0871687b 08702770 081a892e 081aa48f 08064349 08172ea1 086d1120

081a86d7 0817d9a0 08716571 0844dffb 0845141b dd60ec47 08065bf2 087188cf

0805aa94 080600a6 0933b4a1 0933b958 0804cc7d 08768c02 08716866 0805aa94

0933ba03 081945d7 08064499 0876952b 0817d954 0933b8dc 08c3a689 086de88f

0844dffb 0817fef2 081a8687 dd60ec47 0844017a 086e66eb 08714b34 dd60ec47

08c35df1 087167d1 08c4f8f6 081a873f 09348550 dd60ec47 08052180 086e67b8

08764b91 081a87a8 086c66b9 08172ec9 080600cb dd60ec47 086d15a1 086c9bf1

086debb3 08c3744e 0817d954 0817d900 080600d3 08c4f60b 08715ba0 080eb19d

086e00c3 08445a5f 08700561 081a87f0 0871bdeb 087b5095 0843f6f0 dd60ec47

086dff20 086c1080 08716541 086e66c0 08acf4d7 dd60ec47 080522f0 08718adf

0844e010 0805aa94 0843f6e3 087b8969 08700dc9 080523c6 08445ad0 08445a4d

086c85d0 087000e0 0844dffb 08c501cb 0819449a 081a892e 08717cfa 08170de5

08064af9 081aa56e 0871bf8c 086cdbcb 08064af2 081aa463 08183ef9 08767b99

08717345 086ca8ba 081aa48f 08716e8d 0843f6c9 dd6a92c1 0843f82d 0844dffb

08737d6f 08445a53 08c4f79a 0933b961 081aa460 081a8687 0875dd7d 0875922b

08c3744e 086ffa30 081a88ea 0843f9d1 08170ff1 081a86d3 081aa558 08714c08

0805aa94 08194652 086c36c9 08c4f8f6 0818eaab 086defa8 093472fe dd60ec47

086ffdf1 08759404 08767b99 086d1ee4 084420d9 08052170 08716f67 08715bc0

08442ac3 0933b8b0 0933ba2b 0873c020 08716f67 086c9ba8 086c2240 08753fe5

086c9111 08171055 dd6a92dc 08c35d7b 08716e96 dd60ec47 08717c1d 08051fb7

081946c3 081a892e 086ffee2 0818a3a4 085d159e 0843f6e3 08c35abe 0871bc00

0871687b 08170f11 08445a3f 08170ff7 08445b71 dd60ec47 08767b41 081a87fd

086d20c2 093472f0 086dff20 08718113 08c4f6a0 086d20c2 08702562 084401a7

0871890f 08c35e37 08442ad3 08194b60 08194afe 086e6688 08194bca 08c35aff

087680d5 086ffe7d 08718e45 0806434c 081738e8 08c3b000 08064246 08445a67

081a8922 08064564 0822b220 081a8925 dd60ec47 0844e511 086debb3 086dff20

08194b27 086c968a 08716f67 087185b2 dd60ec47 08064af7 086debb3 0844dffb

0818eaab 086ccb45 0874e7bd 08c4f9f0 0843f9b1 085d2184 0843f6d5 08442ad3

08442ad3 081ab900 087cac01 dd60ec47 084420fb 08c4f9a4 081a8687 08716e8d

0818ea9e 08445b60 081a87c1 09364788 08715d6e 0843f6f2 081aa58c 0805f5bb

0817d954 08716f67 08183ef3 08194cc5 08442ad3 0871687b 086c9be4 08064366

0843f6c0 0844e511 08700ea4 08717bd5 08052103 0843f841 084401a4 08194387

087167a0 09364786 0871bc00 dd60ec47 0844dffb 0817d9bb 08c4f68f 0818ea9e

080642c3 08715d82 08445b58 08172ec9 0817fe8f 081892e1 0933b9f9 08c4f653

0843fba1 dd60ec47 086c94b9 08172dff 084502d0 086e66d9 0817d9bb 08055a80

0806428c 081aa558 086debb3 086c6662 0806430f 08183e51 081a894a 080642cf

0871bc03 0871689f 085d2191 08715b81 0844dffb 086ffe5e 0843f6c0 08442ad3

081a87de 0871bba1 087168a5 0817feb4 08715b81 0805aa94 08c34e37 08717bd8

08715bc0 dd60ec47 080523e1 081aa460 08172eae dd60ec47 0817fef2 08445b65

087166d3 08c35d37 0936475b 086d17d0 0876952b dd60ec47 08445a59 0843f6d5

0875991e 08172ec9 08172eae 08714e60 08c35d92 0843f6e3 0817fe81 dd60ec47

080642d5 08450281 08715d4d 0817fef2 08172df4 086c6682 08737d6f 080523a8

0805240c 0816ffb9 081a87c1 0806478b 08737dd1 085d21bd 08c501cb 087c7eb0

081aace4 08737d6f 0817fef2 dd60ec47 08064246 08445a30 08c4f6a2 08c4f8ec

08759bc7 081aa54f 087644f0 08172eae 080521a3 08c34e37 0871bbd7 081951a2

081a86cb dd60ec47 08c38422 08172ec9 08194d26 081a8687 0817d9bb 0874e90d

dd60ec47 08445a42 08c34b16 08442af1 08064315 0933b8b0 08717bd8 08717d88

08767ee5 085d15e8 080642de 08c4f653 086ffe5e 08064315 08170d90 085d159e

0845141b 086c6654 0871c710 0806444d 08172ec9 08445a30 08c38200 08172e9e

0890b560 0843f6cf 081aa860 08194bca 08703603 08718bc8 08442bf9 086ffe5e

08c35150 08702386 08064246 08716625 08c34e82 081ab900 08c363fd 08172e9e

081aa58a 0805aa94 084401bd 08c37d5f 087607ec 0871bba0 08716856 08716f67

08718971 0843f6c0 081ab1c9 08172ec9 08172dff 0844dffb 0819435d 0875967c

085d1591 08170f0e 08065bff 081aa532 08714afe 08c398fd 08718afd 0817d9bb

08759404 08172ec9 08c3b190 086c56ba 081951b3 087652ad 0933b9ff 0875cf7e

08194b60 08064274 08715bc0 081aa466 08441a95 0817fe81 08717bd8 08172eae

08055d93 dd60ec47 0875c63c 086d1316 0844017a 0933b915 08718afd 0875dd00

08c4f804 0817d9bb 081aa545 086c87e8 087c0aae 081ab91a 0804cc7d 08445b7f

081a8730 087188d7 08170d70 086debb3 0805aab1 0804cc8f dd60ec47 081ab901

0933b96d 087682a0 08714836 0804eab0 0844dffb 081aa54c 08716f67 0875977e

08702599 0875c3a1 08716713 087607d3 086def53 0933b9f7 08717bd8 09348550

0805f5bb 0817fef2 0817fe81 086e65d3 080642b1 080522f0 08172ec9 08718113

08c4f6a0 080642af 08064d77 08064672 dd60ec47 08445a4d 086c9b9f 08753ef0

08715bc9 08c5010e 0874e7bd 08064685 0844dffb 086c35a0 0933b961 0933b961

08442adc 081aace4 0845141b 08c391d5 0871c48e 08a6bb90 08442174 080521aa

0843f6e3 081aa479 08170d95 08759f60 dd6a9191 084401a3 0805aa94 08195289

08716e8d 086d2443 081aa492 0817fe81 0843f841 08194811 0805aa94 0804cc74

08172eae 081a874d 08064246 0873be90 08441a92 0843f6e3 08442b4a 0804eab0

08445abb dd60ec47 087188cf 08a5edd6 0875fe2a 08172ec9 0875c0f1 0843f6fe

08172ec9 08442acb 081ab925 0817fefb 08445a31 08445ad7 08737d5b 087b5180

086c87c0 081a8687 0874d2e4 08a5edd6 08440220 0806439d 08c4f8f8 0817fe81

0843f6e3 0871c717 08c375bb 081a87ad 0844dffb 08c4f9f6 0871c726 0844dffb

081aace4 0817d900 08715bc9 dd60ec47 080521b1 08737d6f dd60ec47 08717bd5

dd60ec40 080642c3 0819daf8 081a86cb 086d1f31 0818ea9b dd60ec47 0875a5db

080642af 08715b40 0875d590 08759f60 08440159 08715daf 080521e2 08c5010e

08172ebb 08174280 08c367ab dd60ec47 081a8922 081aa584 08767b41 0805aa94

086deee1 08064246 dd60ec47 08445a30 dd60ec47 086ffe5e 08c3b190 08737d65

08c50167 086c6662 08064ad1 08716e8d 081a86cb dd60ec47 08229d2e 087172d6

0818ea9e 0822b220 08702550 08c4f653 08c34e37 081a8687 0843f6e3 dd60ec47

08172eae 08445a36 08c3b000 08716f67 086c87e8 081ab90c 0817d900 08a5edd0

08172ec9 085d1f90 dd60ec47 0804ddd9 0875dd05 08718611 086e026e 08760e16

dd60ec47 08172e03 0843f6d2 08064246 08064a6a dd60ec47 0804ddec 08445a70

08c351d2 08c365c4 086debb3 081ab19b 08440174 086c92c0 08c4f65b 0843f6e3

08718ef9 084420fb 08715c72 0843f6d2 086c9bf1 0805aa94 081945bd 08c376e2

0843f6dc dd60ec47 0805aa94 086c88d4 0805aa94 08442aa9 080641ec 093472f0

087166b9 081aa492 0818ea9e 08717e30 08717bd8 08702340 081ab925 08445ab0

080600b8 08767c0a 08c36e40 0933b96d 086ffe54 087753b0 081a8687 08445a70

081aa58a 08c35725 dd6aa66f 0805aa94 087b518e 08717b6b 080600b8 08183e73

dd60ec47 0817fef2 081ab1c9 08195289 0805aa94 0805aa94 0875b609 081944a0

086c3694 0875b4d0 086c3590 08716f67 08440167 0844017a 08064299 08194bca

dd60ec47 0845141b 08c4f653 0805220a 0933b961 0844dffb 093472fe 0843f6d2

086c1350 086d1eb0 087166ca 0843f825 087b8969 08c4f6f1 080642d5 dd6a67d3

dd60ec47 08445b6e 0818ea9e dd60ec47 08c3b190 0843f6c1 08445ab0 086f337f

085d2164 081a8710 08194355 08194cc5 0871895b 0844dffb 08172eae 081a8687

08445ad7 0805aad0 080642b1 08702340 085d167e 0843f6f0 08768d49 0844dffb

0804cc8f 085d159e 08a63290 081a8683 086de88f 0817d9bb 08170ff7 08717d8f

08442ad3 0805aa94 085d21b4 0843fba1 087b8969 dd60ec47 081ab1bf 0843f6c0

08445a30 08195293 0933b915 08715bcb 086ffe64 081a88ea 081a894a 086c9bf1

08183e73 08440156 08716eac 081a874d 0806443c 08171055 080521a3 0843f6d5

08715b62 08c50100 086ffe5e 0844dffb 081a872e 081aace4 080642af 08717bf9

dd6a91c1 08172ec9 080642aa 0819460e 08442137 087172d1 0844dffb 08c35f92

0810e931 081aa534 08172dff 08767782 08716e9f 081a87b6 086ffe5e 08715bde

08445a36 08051b21 08052302 0805221a 086f3395 0871bddd 0844dffb 0817d93a

0844dffb 0818ea9e 081ab925 08c4f8fd 087596ee 0933b961 0844dffb 08c501cb

0843f6f2 0817fe81 08172eae 08716ee4 08442d9a 086c9b9c dd60ec47 0817d900

086c5c18 081a8689 086ffe5e 081a894a 081a88e3 086ffe5e 0875ff51 0817fe74

08194387 086c55c4 08172eae 086d234b 0818ea9e 08441e2d 0933b909 08064299

0817d9bb 080520fd 081a872e 08715c72 08714830 08702568 08c356c2 080642aa

0817fef2 081a872e 081aa860 dd60ec47 0844021d 0843f6eb 08065bff 084502d0

08c372b0 0817fef2 081aace4 08716f12 08716f67 085d162f 086deb90 08172e82

0844dffb 086c87e8 08445a3c 0843f6c0 086e6688 08445a3c 0875b9f0 086deea0

0817fe77 081aa690 08716f67 08a5edd6 08714ee2 08c4f653 081aa558 08c381f1

08716620 0804cc8f 0817d954 0876051a 08172eae 08718361 086ffe5e 0875faff

085d159e 08716f67 08717bd8 08441e30 08052125 08442bf9 08172ec9 dd6a92dc

085d1671 0843f98d 0843f6dc 0876788d 087185b2 0844dffb 08c4f68f 0843f6c0

08715b69 dd60ec47 081a892b 0936477f 08172ec9 08442174 08442bf9 08751103

08064246 086e67dd 086ffe26 081aa532 086e66c0 080600b8 0817d93a 08c4f67f

080600c2 08064c64 086c9076 dd60ec47 0806460b 081a87db 087047fe 08714ec4

0871bcd4 081aa690 0806434c 0805aa97 08445a5f 086debb3 0871bbb2 081aa558

0817fd29 085d2184 08737d65 08052174 086ffe5e 08445a36 087023aa 085d159e

08715b58 0817fe81 08716e9f 087b4ea0 08c4fb1a 08716650 081a8793 08172eae

08715d7f 08194bc7 08442b64 dd6a91c1 08c37600 dd60ec47 087185d9 086ffe5e

08064af5 08716f67 08716e96 085d158e 081aa466 08445a48 08172eae 0871aca5

08172dff 08172ec9 08172eae 080642aa 08442ad3 08714862 08172eea 08064221

086c2301 081a872e 0933b961 08702568 08c34a29 086c3379 08445b67 dd60ec47

08715b69 081a873f 08172ec9 086ffe5e dd60ec47 080644bc 08064286 08052298

0844dd9f 086c9bc4 08064334 0875c078 08718366 081aa57f 085d159e 0843f6dc

08764b91 085d159e 08c4f68f 08170d90 08052174 086c3745 081aa860 080642b1

08064221 081a871b 081a86d3 0875b3e3 0804ddd9 08065be0 08194387 0843f6e3

08c35a58 085d2184 0804eb1e 086c5055 0871654d 08718ef6 0844df01 08714580

086ffa76 08716e8d 0816ffbf 086e08c9 081ab1b5 08714771 0843f827 0933b88d

08714862 dd60ec47 08737d65 08064af2 080642e4 08441a98 086e0a62 08c500bb

086de839 0871bba0 dd60ec47 08445afd 08172e9e 081a8a7b dd60ec47 08721f16

086e01fd 08065d20 087166a7 0817108d 08715d8f 0875f2e9 08172ec9 dd60ec47

08c5010e dd60ec47 08716e96 08c35d05 dd60ec47 0843f6f0 0804cc8f 08715bc9

0817d9bb 08065d20 08c3984d 086e6798 08172e9e 080521aa 08c4f602 086c92c0

085d2164 08183e50 08702550 0933b87c 08a67fa0 085d1594 081aa56e 081a87b6

0844dffb 081aa690 081aa532 0844021d 081aa532 087171c8 08440150 080522fb

0845028f 081a88e0 081a874d 086fff58 08194d4a 0844dffb 08c36021 087185a1

08441a92 08172ec9 0817105c dd60ec47 081a8687 0805aa94 081a8942 081a8689

08c35a52 084401b0 086ffdf0 08194663 08c35a40 0805aa97 0844dffb 08064246

087150d0 081a8683 086c9bf1 085d2184 08737d5b 0817d9bb 0843f6d2 08172ea1

08440156 0843f6e3 08c4fa0d 0844dffb 08716f67 087598fe 086c967f 08064d61

086debb3 0817d954 081a872e 081a87d6 0871bddd 086c22ff 081a87b6 085d159e

08c36404 080642b1 0805217c 081aa57f 0843f6d2 081aa532 08c3b190 0843f82a

08442272 081a87c1 086c9106 084419e5 086d244a 0817fcfb 081aa563 086c87c0

08737d6f 081838e7 08052243 087188d7 08715bc9 0817d9bb 08c35d05 0844dffb

081a8787 081ab901 0804cc56 08064246 080642db 08064a7e 0805aa94 0844dffb

08442d9a 081ab900 08c4f67f 08c365e5 081ab900 08714580 0933b961 081a8687

08716858 0843f6c6 08172eae 084420d1 08760892 08c35eef 08442ad3 085d159e

08c3b0b0 08064d60 0843f6d5 08194355 08194d4a 08172eae 08c3a11c 08717bd8

08c4f619 08442117 0819460e 08c381f0 0844dffb 086ffee8 08737d6f 08717bf9

0844dffb 08172eae 08c4f75b 0805aa94 08760a7b dd60ec47 081aa532 081a8687

0816ffac 09347355 08172ec9 08c39297 08445a30 dd60ec47 081a883b dd60ec47

08717bd5 08715bc9 080642b1 080642b1 dd60ec47 08170f26 0933b87c 081aa54f

08172ec9 0874fdf0 08714580 08702589 0843fba1 0843f6dc 08194bca 080642b1

08442d9a 08189337 08716f67 086c1080 084420d1 086c87c0 0843f6c0 08c4f67f

081a88fb 086ffe64 081ab900 081a8801 08194bca 08717bf9 086ca8d1 0844dffb

08440167 0871689f 080523cb 081a872e 08172dff 08721f00 08064309 080600bd

081a872e 0875bd8c 086e68d1 086c66b9 087607ec 086ffdf0 08442ad3 081aa532

086c5c7f 0844dffb 0805aa94 086c88d0 dd60ec47 085d2191 08716e93 0843f6e3

08c4f602 084420cb 0844dffb 081a894d 081a8687 08174a41 0843f6c0 08445a5f

0818a3a4 0875b4e8 086f3370 0875a5c0 08714585 08c4fffe 08445a5f 0817d95e

0819460e 08716744 0817d954 0843f6d7 0933b961 08440247 087021c6 08c4f6e7

08194cc5 0844dffb 081a872e 0844019f 0844dffb 0804cc82 08170f01 0873c020

086fff4e 080523a8 081a87f0 08715b7b 081aa860 0845141b 080642bd 080647ab

08445a3f dd60ec47 0875d590 08172ef6 08171018 0843f6e9 086d1eb0 08715b6c

08445ab0 086c9066 087145b2 08c4f67f 08442d9a 081aa492 080642b1 08172ea1

08acf420 087b5020 0817fef2 08718b85 08716e8d 08c356d6 086ffe7d dd60ec47

081a8904 08c35c37 0818ea9e 08c35ac5 08c35c37 dd60ec47 08172ec9 0804d67a

0870277d 0817d954 dd60ec47 081ab196 080644a6 08717b7f 08442bec 081a8732

0804c964 081a872e 0875fe84 08445a36 08737d5f 0843f6e3 086e49f0 086c85ff

08714581 086c2c38 0933b9f7 087606ff dd60ec47 08c35eef 081a87b6 086c9bf1

081a88fb 08064d86 0844e037 08717bf9 08c4f688 081a8a7e 0844dffb 08c36430

dd60ec47 0844e4f1 0843f6e3 081944c8 08716fff 086e0e48 081ab925 08a6bb7e

081952fb 0844dffb 086c2240 08715b69 0843f6c4 08c372b0 08052195 08172ec9

08442bf9 08195289 08acf40a 0843f6e3 086d2115 0843f6e3 081a88e0 081944a0

0804cc8f 085d1eb0 0876952b 087150d0 087697e0 08767c0a 08172ea1 0843f6e3

0871bcbf 08065c14 0890b560 08064246 081aa49c 08064af6 0843f6c0 0844dff3

08719ef3 0844dffb 086ffe7d 0817fe81 08445a30 0843f6e3 08715b40 081a8683

081a8801 086d21b9 08172ec9 08c3a96a 08055a97 0843f993 08702586 081a86d3

08c4f688 085d1eb0 0845142c 0844215d 08743b83 08172e9e 087c0aa0 08445af0

08052180 08717016 08172ec9 086c9116 086c9c17 dd60ec40 0843f6e3 08194355

0871bba0 086ffe5e 0933b961 086c9066 08737da5 08716e30 08172ec9 085d2184

081aa57f 08721f00 08442e37 0817d9b3 0843f957 08716f67 0817fef2 081945d7

081749e1 0843f6e9 0890b560 0871aa80 085d165f 081ab925 08c35d05 08737d35

080643e6 dd60ec47 0817fe31 08172ec9 0936474c 081a8912 084401a9 0870070e

08442174 08172eae 081aa479 08c35c80 08737d65 08716e8d 0805aa94 081a8901

0817d9bb 086c1351 0844dffb 08716f67 0843f6d2 0844dffb 08715bc9 08445a70

08715d93 084420d3 081a8710 08704c40 081a872e 0844dffb 0844dffb 0805aa94

08716eac 081a892e 0804ddd9 08715bc9 086debb3 08172ec9 08c4f602 081aace4

08441e0d 081a872e 0843f6e9 0875abf6 0843fbac 08064246 081aa460 08716e8d

0875922b 08442af6 08a5edce 085d167e 0804cc72 0843f6eb 0806444d 08c3825c

081a872e 08172eae 081ab901 087b521a 081a88fb 08172ec9 0844dffb 0843f6e3

081aa57f 087172ce 0871655d 081a88ec 08172e07 086c87e8 08c3b2f0 087697e0

084502d0 085d2184 0875fa90 08442ae8 08c4f723 0844dffb 0806009f 08172ef6

08717bd8 08442ad9 086e6783 081a8687 08171063 087b8969 086c3057 08055a97

084420fb 087023a0 08c35e2c 08183ef9 08172ea1 083ec130 086c88d0 081735f8

08445b76 08442ad3 08759139 0843fbd4 0843f6d7 081aa532 08c4fee6 0844dffb

08716f67 0844defe 0843f6d5 08195ef4 08055a97 08716782 08718ef6 0844dffb

086c87e8 0933b8b0 0844dffb 08064430 087c0aa0 08052180 0890b560 08c372b0

086c9bf5 08704a60 0844dffb 08445acb 08172ec9 08714830 081aace0 081ab926

08714784 08170fd9 0844e4fd 087188d9 08717bd5 0843f6e3 087753b1 08445a53

081a87f0 081a8787 08717f19 086e0ead 08716fd4 086e6783 dd60ec47 dd60ec47

08716796 086debb3 08718afd 08c4f602 0871485b 08717345 08064020 08c3743c

087b8969 08194cc5 080520eb 08064315 0805aac1 085d159e 0817d9bb 08172ec7

0843f6e3 08183840 08717bd8 087682a0 08064d60 081a87de 081a892b 08442bf9

0843f6e3 0933b9f9 08052340 0845141b 086e66ce 081a872a dd60ec47 08440224

0843f6c0 081aa56e 081a87c1 081aa563 0843f6e3 08172eae dd60ec47 dd60ec47

0844e4f1 0843f900 086e6525 08c4f6a0 0804cc8f dd60ec47 0805aab1 08064a7e

0871bba0 dd60ec47 086c6562 08064a7e 085d2191 080642a5 0844dffb 08445b67

086c43e0 08172e9e 081aa532 080600bd 081a892e 086e03ff 0871aa13 080642db

08c4ffde 08716780 08442dc8 086d21ae 08715bc9 081735f8 081a872e 08737d6f

081838e7 08716e8d 08064246 081aa697 dd60ec47 08715bc0 086df2b9 081ab926

087025a7 08716780 dd60ec47 08064283 080600aa 0844dffb 0871aca1 0805aa94

dd60ec47 08172ec9 0817d9bb 080643a3 081ab926 08759481 0843f6e3 0843f6d2

0843f6d2 08442adc 0817d954 081ab900 086e49f0 09346501 085d159e 08445a30

086c8580 08445a30 08441db0 0817fd29 08718afd 09340c5a 0934597c 09348571

080642b1 087bf00b 08c35c37 08064c6b 08445aeb 080642b1 08716e78 087147a1

08441e3e 085d2100 081a86d3 08717c1d 08714b2a 08172ec9 08702384 0819477f

08172eae 0843f6e3 0844dffb 08170fa2 081952cb 08194340 08170f24 dd60ec47

08c36278 081a8935 08064af5 08194355 0817fe81 0843f6e3 0844dffb 087b8969

08445a42 08717bd5 08064394 08716ea3 08c370c0 086e49fd 08716780 081944a0

08442c5f 081944a4 086c3699 086deee1 0805aa94 08718130 085d2100 08172eae

08172ec9 08c3513e dd6a918e 08717bf9 08a6bba0 08716f67 086dff53 081a87b6

08064520 08172dff 08445a30 08715bc0 dd6a690e 081a892b 081a88f4 0933b961

08716fff 08759bc7 08194cc5 dd60ec47 0933ba03 086c663b 081a8ab0 0871c710

0933b431 0871484c 08194cdeEnd of samples

i-ASA-5520# clear cpu profile dump

                                  ^

ERROR: % Invalid input detected at '^' marker.

i-ASA-5520# clear cpu profile

i-ASA-5520# show conn count

10081 in use, 23624 most used

i-ASA-5520# sh xlate count

1779 in use, 26962 most used

i-ASA-5520# sh cpu usage

CPU utilization for 5 seconds = 75%; 1 minute: 74%; 5 minutes: 72%

i-ASA-5520#

Joe B Danford Thu, 05/23/2013 - 06:13

Thanks for the output. You have quite a bit of overruns on your inside and outside interfaces. We need to determine if this is actively happening  or if its old data. Can you collect the following?

show traffic

show clock

show proc cpu-hog


Following output, 3 times 1 minute apart

show int | i overrun

show process cpu-usage sorted non-zero
show asp drop

To help mitigate the overruns you can consider enabling flow control also

interface GigabitEthernet0/0

     flowcontrol send on

interface GigabitEthernet0/1

     flowcontrol send on

More info on flow control can be found here:

http://www.cisco.com/en/US/docs/security/asa/asa82/configuration/guide/intrface.html

Mokeev.tel Thu, 05/23/2013 - 09:26

Is

flowcontrol send on

safe on high loaded device?

I'll read about it.

Here is output:

i-ASA-5520# show traffic

outside:

        received (in 908406.520 secs):

                65042364731 packets     11631112108662 bytes

                71000 pkts/sec  12803001 bytes/sec

        transmitted (in 908406.520 secs):

                60955497150 packets     11000835610826 bytes

                67002 pkts/sec  12110002 bytes/sec

      1 minute input rate 28534 pkts/sec,  4941268 bytes/sec

      1 minute output rate 26639 pkts/sec,  5158941 bytes/sec

      1 minute drop rate, 3 pkts/sec

      5 minute input rate 32477 pkts/sec,  6828890 bytes/sec

      5 minute output rate 29197 pkts/sec,  6200496 bytes/sec

      5 minute drop rate, 3 pkts/sec

inside:

        received (in 908406.550 secs):

                62510613124 packets     11133255889575 bytes

                68000 pkts/sec  12255003 bytes/sec

        transmitted (in 908406.550 secs):

                66539099130 packets     11749698861952 bytes

                73002 pkts/sec  12934003 bytes/sec

      1 minute input rate 27375 pkts/sec,  5274791 bytes/sec

      1 minute output rate 29265 pkts/sec,  5056875 bytes/sec

      1 minute drop rate, 1 pkts/sec

      5 minute input rate 30002 pkts/sec,  6355307 bytes/sec

      5 minute output rate 33278 pkts/sec,  6983478 bytes/sec

      5 minute drop rate, 1 pkts/sec

management:

        received (in 908407.870 secs):

                150536 packets  6924656 bytes

                0 pkts/sec      2 bytes/sec

        transmitted (in 908407.870 secs):

                398901 packets  11685908 bytes

                0 pkts/sec      3 bytes/sec

      1 minute input rate 0 pkts/sec,  0 bytes/sec

      1 minute output rate 0 pkts/sec,  0 bytes/sec

      1 minute drop rate, 0 pkts/sec

      5 minute input rate 0 pkts/sec,  0 bytes/sec

      5 minute output rate 0 pkts/sec,  0 bytes/sec

      5 minute drop rate, 0 pkts/sec

failover:

        received (in 908407.900 secs):

                32388040 packets        3996205496 bytes

                2 pkts/sec      4001 bytes/sec

        transmitted (in 908407.900 secs):

                32389587 packets        3893781076 bytes

                2 pkts/sec      4002 bytes/sec

      1 minute input rate 1 pkts/sec,  171 bytes/sec

      1 minute output rate 1 pkts/sec,  177 bytes/sec

      1 minute drop rate, 0 pkts/sec

      5 minute input rate 1 pkts/sec,  172 bytes/sec

      5 minute output rate 1 pkts/sec,  179 bytes/sec

      5 minute drop rate, 0 pkts/sec

state:

        received (in 908408.470 secs):

                3193200737 packets      3603824483592 bytes

                3004 pkts/sec   3967000 bytes/sec

        transmitted (in 908408.470 secs):

                574490212 packets       638211330526 bytes

                3 pkts/sec      702001 bytes/sec

      1 minute input rate 1 pkts/sec,  94 bytes/sec

      1 minute output rate 221 pkts/sec,  255097 bytes/sec

      1 minute drop rate, 0 pkts/sec

      5 minute input rate 1 pkts/sec,  94 bytes/sec

      5 minute output rate 222 pkts/sec,  255414 bytes/sec

      5 minute drop rate, 0 pkts/sec

----------------------------------------

Aggregated Traffic on Physical Interface

----------------------------------------

GigabitEthernet0/0:

        received (in 908408.500 secs):

                65042430038 packets     13034162799392 bytes

                71004 pkts/sec  14348004 bytes/sec

        transmitted (in 908408.500 secs):

                60954769586 packets     12210338269704 bytes

                67001 pkts/sec  13441003 bytes/sec

      1 minute input rate 28533 pkts/sec,  5557154 bytes/sec

      1 minute output rate 26638 pkts/sec,  5684850 bytes/sec

      1 minute drop rate, 0 pkts/sec

      5 minute input rate 32477 pkts/sec,  7516782 bytes/sec

      5 minute output rate 29197 pkts/sec,  6781095 bytes/sec

      5 minute drop rate, 0 pkts/sec

GigabitEthernet0/1:

        received (in 908408.980 secs):

                62510686243 packets     12378277109614 bytes

                68000 pkts/sec  13626004 bytes/sec

        transmitted (in 908408.980 secs):

                66539163351 packets     13186194711545 bytes

                73002 pkts/sec  14515002 bytes/sec

      1 minute input rate 27375 pkts/sec,  5816267 bytes/sec

      1 minute output rate 29266 pkts/sec,  5689262 bytes/sec

      1 minute drop rate, 0 pkts/sec

      5 minute input rate 30002 pkts/sec,  6952684 bytes/sec

      5 minute output rate 33278 pkts/sec,  7688082 bytes/sec

      5 minute drop rate, 0 pkts/sec

GigabitEthernet0/2:

        received (in 908409.010 secs):

                3225588779 packets      3742000376778 bytes

                3002 pkts/sec   4119001 bytes/sec

        transmitted (in 908409.010 secs):

                606879922 packets       655456745262 bytes

                1 pkts/sec      721004 bytes/sec

      1 minute input rate 2 pkts/sec,  322 bytes/sec

      1 minute output rate 223 pkts/sec,  260260 bytes/sec

      1 minute drop rate, 0 pkts/sec

      5 minute input rate 2 pkts/sec,  324 bytes/sec

      5 minute output rate 223 pkts/sec,  260512 bytes/sec

      5 minute drop rate, 0 pkts/sec

GigabitEthernet0/3:

        received (in 908409.150 secs):

                0 packets       0 bytes

                0 pkts/sec      0 bytes/sec

        transmitted (in 908409.150 secs):

                0 packets       0 bytes

                0 pkts/sec      0 bytes/sec

      1 minute input rate 0 pkts/sec,  0 bytes/sec

      1 minute output rate 0 pkts/sec,  0 bytes/sec

      1 minute drop rate, 0 pkts/sec

      5 minute input rate 0 pkts/sec,  0 bytes/sec

      5 minute output rate 0 pkts/sec,  0 bytes/sec

      5 minute drop rate, 0 pkts/sec

Internal-Control0/0:

        received (in 908409.190 secs):

                167433701 packets       17308399938 bytes

                4 pkts/sec      19001 bytes/sec

        transmitted (in 908409.190 secs):

                172878249 packets       12198529951 bytes

                1 pkts/sec      13002 bytes/sec

      1 minute input rate 6 pkts/sec,  442 bytes/sec

      1 minute output rate 6 pkts/sec,  458 bytes/sec

      1 minute drop rate, 0 pkts/sec

      5 minute input rate 6 pkts/sec,  442 bytes/sec

      5 minute output rate 6 pkts/sec,  458 bytes/sec

      5 minute drop rate, 0 pkts/sec

Internal-Data0/0:

        received (in 908409.320 secs):

                11815371242 packets     2089587494735 bytes

                13001 pkts/sec  2300001 bytes/sec

        transmitted (in 908409.320 secs):

                11819736481 packets     2091293181857 bytes

                13002 pkts/sec  2302001 bytes/sec

      1 minute input rate 3 pkts/sec,  304 bytes/sec

      1 minute output rate 3 pkts/sec,  304 bytes/sec

      1 minute drop rate, 0 pkts/sec

      5 minute input rate 3 pkts/sec,  305 bytes/sec

      5 minute output rate 3 pkts/sec,  305 bytes/sec

      5 minute drop rate, 0 pkts/sec

Management0/0:

        received (in 908409.460 secs):

                150421 packets  9032160 bytes

                0 pkts/sec      0 bytes/sec

        transmitted (in 908409.460 secs):

                398432 packets  25792376 bytes

                0 pkts/sec      0 bytes/sec

      1 minute input rate 0 pkts/sec,  0 bytes/sec

      1 minute output rate 0 pkts/sec,  0 bytes/sec

      1 minute drop rate, 0 pkts/sec

      5 minute input rate 0 pkts/sec,  0 bytes/sec

      5 minute output rate 0 pkts/sec,  0 bytes/sec

      5 minute drop rate, 0 pkts/sec

i-ASA-5520#

i-ASA-5520# show clock

13:13:03.619 EDT Thu May 23 2013

i-ASA-5520# show proc cpu-hog

Process:      rtcli async executor process, PROC_PC_TOTAL: 1, MAXHOG: 2, LASTHOG: 2

LASTHOG At:   14:14:23 EDT Apr 30 2013

PC:           8c73c5d (suspend)

Process:      rtcli async executor process, NUMHOG: 1, MAXHOG: 2, LASTHOG: 2

LASTHOG At:   14:14:23 EDT Apr 30 2013

PC:           8c73c5d (suspend)

Call stack:   8c73c5d  8c564ca  8c58f86  8c59035  37f  7118562c

Process:      ssh, PROC_PC_TOTAL: 1, MAXHOG: 6, LASTHOG: 6

LASTHOG At:   21:48:08 EDT Apr 30 2013

PC:           8a6a377 (suspend)

Process:      ssh, NUMHOG: 1, MAXHOG: 6, LASTHOG: 6

LASTHOG At:   21:48:08 EDT Apr 30 2013

PC:           8a6a377 (suspend)

Call stack:   8a6a377  88e719b  89a8c2c  88d85b2  88a5c16  88a733a  80cb231

              80cc58a  80cd30a  8063ba3

Process:      ssh, PROC_PC_TOTAL: 1, MAXHOG: 5, LASTHOG: 5

LASTHOG At:   21:48:08 EDT Apr 30 2013

PC:           83b3ff7 (suspend)

Process:      ssh, NUMHOG: 1, MAXHOG: 5, LASTHOG: 5

LASTHOG At:   21:48:08 EDT Apr 30 2013

PC:           83b3ff7 (suspend)

Call stack:   84831ee  8954a42  80e9e03  89a8f43  88d85b2  88a5c16  88a733a

              80cb231  80cc58a  80cd30a  8063ba3

Process:      ssh, PROC_PC_TOTAL: 1, MAXHOG: 2, LASTHOG: 2

LASTHOG At:   16:44:53 EDT May 6 2013

PC:           889cd25 (suspend)

Process:      ssh, NUMHOG: 1, MAXHOG: 2, LASTHOG: 2

LASTHOG At:   16:44:53 EDT May 6 2013

PC:           889cd25 (suspend)

Call stack:   889cd25  80d84fc  80d8d96  80cc411  80cd30a  8063ba3

Process:      fover_health_monitoring_thread, PROC_PC_TOTAL: 1, MAXHOG: 3, LASTHOG: 3

LASTHOG At:   13:31:59 EDT May 14 2013

PC:           83ccbfc (suspend)

Process:      fover_health_monitoring_thread, NUMHOG: 1, MAXHOG: 3, LASTHOG: 3

LASTHOG At:   13:31:59 EDT May 14 2013

PC:           83ccbfc (suspend)

Call stack:   83ccbfc  8063ba3

Process:      ssh, PROC_PC_TOTAL: 1, MAXHOG: 2, LASTHOG: 2

LASTHOG At:   03:54:04 EDT May 22 2013

PC:           8c2c79b (suspend)

Process:      ssh, NUMHOG: 1, MAXHOG: 2, LASTHOG: 2

LASTHOG At:   03:54:04 EDT May 22 2013

PC:           8c2c79b (suspend)

Call stack:   8c2cb92  8bde70c  8bed5e2  8be31f5  8be338f  8be34c3  889ce34

              935f511  8487081  84969cb  8958682  88a5c16  88a733a  80cb231

Process:      fover_parse, NUMHOG: 133, MAXHOG: 7, LASTHOG: 3

LASTHOG At:   13:52:36 EDT May 22 2013

PC:           8064d55 (suspend)

Call stack:   8064d55  88de0db  88de2af  88de2f8  88e0bfb  88a4bbf  889dd70

              897a14c  897a211  848eab7  84850be  848a6b2  8494dca  8958c2a

Process:      rtcli async executor process, PROC_PC_TOTAL: 12, MAXHOG: 5, LASTHOG: 2

LASTHOG At:   15:36:12 EDT May 22 2013

PC:           8c562db (suspend)

Process:      rtcli async executor process, NUMHOG: 12, MAXHOG: 5, LASTHOG: 2

LASTHOG At:   15:36:12 EDT May 22 2013

PC:           8c562db (suspend)

Call stack:   8c562db  8c58f86  8c59035  37f  7118562c

Process:      ssh, PROC_PC_TOTAL: 27308, MAXHOG: 9, LASTHOG: 3

LASTHOG At:   16:34:37 EDT May 22 2013

PC:           8bde96c (suspend)

Process:      ssh, NUMHOG: 1195, MAXHOG: 9, LASTHOG: 3

LASTHOG At:   16:34:37 EDT May 22 2013

PC:           8bde96c (suspend)

Call stack:   8bde96c  8bed5e2  8be31f5  8be338f  8be34c3  889ce34  935c0d8

              80d860d  80d8e35  80cc411  80cd30a  8063ba3

Process:      Dispatch Unit, PROC_PC_TOTAL: 15281, MAXHOG: 27, LASTHOG: 3

LASTHOG At:   00:13:13 EDT May 23 2013

PC:           81a88af (suspend)

Process:      Dispatch Unit, NUMHOG: 15246, MAXHOG: 27, LASTHOG: 3

LASTHOG At:   00:13:13 EDT May 23 2013

PC:           81a88af (suspend)

Call stack:   81a88af  8063ba3

Process:      Dispatch Unit, PROC_PC_TOTAL: 468632, MAXHOG: 32, LASTHOG: 3

LASTHOG At:   00:13:13 EDT May 23 2013

PC:           81a8999 (suspend)

Process:      Dispatch Unit, NUMHOG: 454098, MAXHOG: 32, LASTHOG: 3

LASTHOG At:   00:13:13 EDT May 23 2013

PC:           81a8999 (suspend)

Call stack:   81a8999  8063ba3

Process:      ssh_init, NUMHOG: 83, MAXHOG: 8, LASTHOG: 7

LASTHOG At:   06:52:29 EDT May 23 2013

PC:           8064d55 (suspend)

Call stack:   8064d55  8136c2b  92e4fdc  92f9d4f  92e74ad  92e7147  92e6f18

              92d641f  92c5e6c  92e61b2  92e6029  92d5166  92c5c1c  812fedf

Process:      ssh_init, PROC_PC_TOTAL: 62, MAXHOG: 9, LASTHOG: 8

LASTHOG At:   06:52:29 EDT May 23 2013

PC:           83dc173 (suspend)

Process:      ssh_init, NUMHOG: 62, MAXHOG: 9, LASTHOG: 8

LASTHOG At:   06:52:29 EDT May 23 2013

PC:           83dc173 (suspend)

Call stack:   83dcced  83d5903  812ff8c  8a1be92  89ed1ea  8be4cc4  8be40f8

              8063ba3

Process:      ssh_init, PROC_PC_TOTAL: 19, MAXHOG: 3, LASTHOG: 3

LASTHOG At:   06:52:30 EDT May 23 2013

PC:           806f805 (suspend)

Process:      ssh_init, NUMHOG: 19, MAXHOG: 3, LASTHOG: 3

LASTHOG At:   06:52:30 EDT May 23 2013

PC:           806f805 (suspend)

Call stack:   8be1156  8bef867  8be40ca  8063ba3

Process:      fover_parse, PROC_PC_TOTAL: 445, MAXHOG: 34, LASTHOG: 7

LASTHOG At:   06:52:30 EDT May 23 2013

PC:           8064d55 (suspend)

Process:      ssh_init, NUMHOG: 20, MAXHOG: 8, LASTHOG: 7

LASTHOG At:   06:52:30 EDT May 23 2013

PC:           8064d55 (suspend)

Call stack:   8064d55  8beef69  8be4043  8063ba3

Process:      Unicorn Admin Handler, PROC_PC_TOTAL: 776, MAXHOG: 14, LASTHOG: 3

LASTHOG At:   12:52:42 EDT May 23 2013

PC:           88a4d45 (suspend)

Process:      Unicorn Admin Handler, NUMHOG: 776, MAXHOG: 14, LASTHOG: 3

LASTHOG At:   12:52:42 EDT May 23 2013

PC:           88a4d45 (suspend)

Call stack:   88a4d45  889dd70  897a14c  897a211  92ff8a3  89b8544  897a211

              889dbe7  889dc51  88a5c16  88a733a  80cb231  84372b0  842fcee

CPU hog threshold (msec):  2.844

Last cleared: None

i-ASA-5520#

i-ASA-5520# show int | i overrun

        26883089 input errors, 0 CRC, 0 frame, 26883089 overrun, 0 ignored, 0 abort

        31799032 input errors, 0 CRC, 0 frame, 31799032 overrun, 0 ignored, 0 abort

        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

i-ASA-5520# show process cpu-usage sorted non-zero

PC         Thread       5Sec     1Min     5Min   Process

081a8999   6c1afa08    66.3%    65.7%    66.6%   Dispatch Unit

08bde96c   6c188600     1.3%     0.2%     0.1%   ssh

i-ASA-5520# show asp drop

Frame drop:

  Invalid TCP Length (invalid-tcp-hdr-length)                               1131

  Invalid UDP Length (invalid-udp-length)                                    607

  No valid adjacency (no-adjacency)                                       456488

  Flow is denied by configured rule (acl-drop)                          25092418

  Invalid SPI (np-sp-invalid-spi)                                           1380

  NAT-T keepalive message (natt-keepalive)                                 74358

  First TCP packet not SYN (tcp-not-syn)                                14728041

  Bad TCP checksum (bad-tcp-cksum)                                         12633

  Bad TCP flags (bad-tcp-flags)                                             4946

  TCP data send after FIN (tcp-data-past-fin)                                587

  TCP failed 3 way handshake (tcp-3whs-failed)                           2414464

  TCP RST/FIN out of order (tcp-rstfin-ooo)                              6849391

  TCP SEQ in SYN/SYNACK invalid (tcp-seq-syn-diff)                          3852

  TCP SYNACK on established conn (tcp-synack-ooo)                              9

  TCP packet SEQ past window (tcp-seq-past-win)                            28713

  TCP invalid ACK (tcp-invalid-ack)                                        46867

  TCP replicated flow pak drop (tcp-fo-drop)                              103179

  TCP ACK in 3 way handshake invalid (tcp-discarded-ooo)                     151

  TCP Out-of-Order packet buffer full (tcp-buffer-full)                    59864

  TCP Out-of-Order packet buffer timeout (tcp-buffer-timeout)             182566

  TCP RST/SYN in window (tcp-rst-syn-in-win)                              630967

  TCP dup of packet in Out-of-Order queue (tcp-dup-in-queue)            22726631

  TCP packet failed PAWS test (tcp-paws-fail)                             143970

  IPSEC tunnel is down (ipsec-tun-down)                                      448

  Slowpath security checks failed (sp-security-failed)                        99

  Expired flow (flow-expired)                                                610

  ICMP Inspect seq num not matched (inspect-icmp-seq-num-not-matched)        767

  ICMP Error Inspect no existing conn (inspect-icmp-error-no-existing-conn)                                     6

  DNS Inspect invalid packet (inspect-dns-invalid-pak)                      9717

  DNS Inspect id not matched (inspect-dns-id-not-matched)                    431

  Interface is down (interface-down)                                        4077

  Dropped pending packets in a closed socket (np-socket-closed)              413

Last clearing: Never

Flow drop:

  NAT failed (nat-failed)                                                     50

  NAT reverse path failed (nat-rpf-failed)                                     2

  Need to start IKE negotiation (need-ike)                                  7682

  Inspection failure (inspect-fail)                                         3912

  SSL handshake failed (ssl-handshake-failed)                                  1

  SSL received close alert (ssl-received-close-alert)                          1

Last clearing: Never

i-ASA-5520# show int | i overrun

        26883089 input errors, 0 CRC, 0 frame, 26883089 overrun, 0 ignored, 0 abort

        31799032 input errors, 0 CRC, 0 frame, 31799032 overrun, 0 ignored, 0 abort

        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

i-ASA-5520# show process cpu-usage sorted non-zero

PC         Thread       5Sec     1Min     5Min   Process

081a8999   6c1afa08    65.0%    65.9%    66.5%   Dispatch Unit

08bde96c   6c188600     0.1%     0.1%     0.0%   ssh

083ccbfc   6c1a17a0     0.1%     0.1%     0.1%   fover_health_monitoring_thread

i-ASA-5520# show asp drop

Frame drop:

  Invalid TCP Length (invalid-tcp-hdr-length)                               1131

  Invalid UDP Length (invalid-udp-length)                                    607

  No valid adjacency (no-adjacency)                                       456488

  Flow is denied by configured rule (acl-drop)                          25092558

  Invalid SPI (np-sp-invalid-spi)                                           1380

  NAT-T keepalive message (natt-keepalive)                                 74364

  First TCP packet not SYN (tcp-not-syn)                                14728114

  Bad TCP checksum (bad-tcp-cksum)                                         12633

  Bad TCP flags (bad-tcp-flags)                                             4946

  TCP data send after FIN (tcp-data-past-fin)                                587

  TCP failed 3 way handshake (tcp-3whs-failed)                           2414466

  TCP RST/FIN out of order (tcp-rstfin-ooo)                              6849423

  TCP SEQ in SYN/SYNACK invalid (tcp-seq-syn-diff)                          3852

  TCP SYNACK on established conn (tcp-synack-ooo)                              9

  TCP packet SEQ past window (tcp-seq-past-win)                            28713

  TCP invalid ACK (tcp-invalid-ack)                                        46867

  TCP replicated flow pak drop (tcp-fo-drop)                              103179

  TCP ACK in 3 way handshake invalid (tcp-discarded-ooo)                     151

  TCP Out-of-Order packet buffer full (tcp-buffer-full)                    59864

  TCP Out-of-Order packet buffer timeout (tcp-buffer-timeout)             182566

  TCP RST/SYN in window (tcp-rst-syn-in-win)                              630968

  TCP dup of packet in Out-of-Order queue (tcp-dup-in-queue)            22726631

  TCP packet failed PAWS test (tcp-paws-fail)                             143974

  IPSEC tunnel is down (ipsec-tun-down)                                      448

  Slowpath security checks failed (sp-security-failed)                        99

  Expired flow (flow-expired)                                                610

  ICMP Inspect seq num not matched (inspect-icmp-seq-num-not-matched)        767

  ICMP Error Inspect no existing conn (inspect-icmp-error-no-existing-conn)                                     6

  DNS Inspect invalid packet (inspect-dns-invalid-pak)                      9717

  DNS Inspect id not matched (inspect-dns-id-not-matched)                    431

  Interface is down (interface-down)                                        4077

  Dropped pending packets in a closed socket (np-socket-closed)              413

Last clearing: Never

Flow drop:

  NAT failed (nat-failed)                                                     50

  NAT reverse path failed (nat-rpf-failed)                                     2

  Need to start IKE negotiation (need-ike)                                  7682

  Inspection failure (inspect-fail)                                         3912

  SSL handshake failed (ssl-handshake-failed)                                  1

  SSL received close alert (ssl-received-close-alert)                          1

Last clearing: Never

i-ASA-5520# show int | i overrun

        26883089 input errors, 0 CRC, 0 frame, 26883089 overrun, 0 ignored, 0 abort

        31799032 input errors, 0 CRC, 0 frame, 31799032 overrun, 0 ignored, 0 abort

        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

i-ASA-5520# show process cpu-usage sorted non-zero

PC         Thread       5Sec     1Min     5Min   Process

081a86c4   6c1afa08    69.8%    67.3%    66.8%   Dispatch Unit

08bde96c   6c188600     0.4%     0.2%     0.0%   ssh

08c50c78   6c19dc90     0.1%     0.0%     0.0%   snmp

i-ASA-5520# show asp drop

Frame drop:

  Invalid TCP Length (invalid-tcp-hdr-length)                               1131

  Invalid UDP Length (invalid-udp-length)                                    607

  No valid adjacency (no-adjacency)                                       456488

  Flow is denied by configured rule (acl-drop)                          25092751

  Invalid SPI (np-sp-invalid-spi)                                           1380

  NAT-T keepalive message (natt-keepalive)                                 74369

  First TCP packet not SYN (tcp-not-syn)                                14728175

  Bad TCP checksum (bad-tcp-cksum)                                         12633

  Bad TCP flags (bad-tcp-flags)                                             4946

  TCP data send after FIN (tcp-data-past-fin)                                587

  TCP failed 3 way handshake (tcp-3whs-failed)                           2414468

  TCP RST/FIN out of order (tcp-rstfin-ooo)                              6849444

  TCP SEQ in SYN/SYNACK invalid (tcp-seq-syn-diff)                          3852

  TCP SYNACK on established conn (tcp-synack-ooo)                              9

  TCP packet SEQ past window (tcp-seq-past-win)                            28713

  TCP invalid ACK (tcp-invalid-ack)                                        46867

  TCP replicated flow pak drop (tcp-fo-drop)                              103179

  TCP ACK in 3 way handshake invalid (tcp-discarded-ooo)                     151

  TCP Out-of-Order packet buffer full (tcp-buffer-full)                    59864

  TCP Out-of-Order packet buffer timeout (tcp-buffer-timeout)             182566

  TCP RST/SYN in window (tcp-rst-syn-in-win)                              631012

  TCP dup of packet in Out-of-Order queue (tcp-dup-in-queue)            22726631

  TCP packet failed PAWS test (tcp-paws-fail)                             143974

  IPSEC tunnel is down (ipsec-tun-down)                                      448

  Slowpath security checks failed (sp-security-failed)                        99

  Expired flow (flow-expired)                                                610

  ICMP Inspect seq num not matched (inspect-icmp-seq-num-not-matched)        767

  ICMP Error Inspect no existing conn (inspect-icmp-error-no-existing-conn)                                     6

  DNS Inspect invalid packet (inspect-dns-invalid-pak)                      9717

  DNS Inspect id not matched (inspect-dns-id-not-matched)                    431

  Interface is down (interface-down)                                        4077

  Dropped pending packets in a closed socket (np-socket-closed)              413

Last clearing: Never

Flow drop:

  NAT failed (nat-failed)                                                     50

  NAT reverse path failed (nat-rpf-failed)                                     2

  Need to start IKE negotiation (need-ike)                                  7682

  Inspection failure (inspect-fail)                                         3912

  SSL handshake failed (ssl-handshake-failed)                                  1

  SSL received close alert (ssl-received-close-alert)                          1

Last clearing: Never

i-ASA-5520#

Julio Carvajal Thu, 05/23/2013 - 09:44

Hello Sergey,

I would like the following information to help as well

show processes cpu-usage sorted non-zero

show interface | include errors

after one minute

show interface | include errors

show traffic

after one minute

show traffic

sh local-host connection tcp 1000 UDP 1000

show local-host connection udp 2000 tcp 2000 ( If you do not get any output stop there, if you still get go for the 3000 and keeps going until you do not receive anything)

Based on the results we will move forward

Keep me posted,

Mokeev.tel Thu, 05/23/2013 - 10:00

Ok,

Here result:

i-ASA-5520# show processes cpu-usage sorted non-zero

PC         Thread       5Sec     1Min     5Min   Process

081a8999   6c1afa08    67.3%    65.9%    65.6%   Dispatch Unit

08bde96c   6c188600     0.1%     0.0%     0.0%   ssh

083ccbfc   6c1a17a0     0.1%     0.1%     0.1%   fover_health_monitoring_thread

i-ASA-5520# show interface | include errors

        26883089 input errors, 0 CRC, 0 frame, 26883089 overrun, 0 ignored, 0 abort

        0 output errors, 0 collisions, 0 interface resets

        31799147 input errors, 0 CRC, 0 frame, 31799147 overrun, 0 ignored, 0 abort

        0 output errors, 0 collisions, 7 interface resets

        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

        0 output errors, 0 collisions, 3 interface resets

        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

        0 output errors, 0 collisions, 0 interface resets

        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

        0 output errors, 0 collisions, 0 interface resets

i-ASA-5520#

i-ASA-5520# show interface | include errors

        26883089 input errors, 0 CRC, 0 frame, 26883089 overrun, 0 ignored, 0 abort

        0 output errors, 0 collisions, 0 interface resets

        31799147 input errors, 0 CRC, 0 frame, 31799147 overrun, 0 ignored, 0 abort

        0 output errors, 0 collisions, 7 interface resets

        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

        0 output errors, 0 collisions, 3 interface resets

        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

        0 output errors, 0 collisions, 0 interface resets

        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

        0 output errors, 0 collisions, 0 interface resets

i-ASA-5520# show traffic

outside:

        received (in 910612.230 secs):

                65108572539 packets     11643688883081 bytes

                71004 pkts/sec  12786004 bytes/sec

        transmitted (in 910612.230 secs):

                61016247971 packets     11012907973607 bytes

                67001 pkts/sec  12093002 bytes/sec

      1 minute input rate 29489 pkts/sec,  5484344 bytes/sec

      1 minute output rate 27129 pkts/sec,  5198507 bytes/sec

      1 minute drop rate, 7 pkts/sec

      5 minute input rate 28705 pkts/sec,  5119003 bytes/sec

      5 minute output rate 26615 pkts/sec,  5056277 bytes/sec

      5 minute drop rate, 3 pkts/sec

inside:

        received (in 910612.260 secs):

                62573068428 packets     11145622712995 bytes

                68003 pkts/sec  12239003 bytes/sec

        transmitted (in 910612.260 secs):

                66607001373 packets     11762569379213 bytes

                73003 pkts/sec  12917002 bytes/sec

      1 minute input rate 27898 pkts/sec,  5327714 bytes/sec

      1 minute output rate 30251 pkts/sec,  5611615 bytes/sec

      1 minute drop rate, 1 pkts/sec

      5 minute input rate 27341 pkts/sec,  5170050 bytes/sec

      5 minute output rate 29427 pkts/sec,  5232482 bytes/sec

      5 minute drop rate, 1 pkts/sec

management:

        received (in 910613.050 secs):

                150536 packets  6924656 bytes

                0 pkts/sec      2 bytes/sec

        transmitted (in 910613.050 secs):

                398901 packets  11685908 bytes

                0 pkts/sec      3 bytes/sec

      1 minute input rate 0 pkts/sec,  0 bytes/sec

      1 minute output rate 0 pkts/sec,  0 bytes/sec

      1 minute drop rate, 0 pkts/sec

      5 minute input rate 0 pkts/sec,  0 bytes/sec

      5 minute output rate 0 pkts/sec,  0 bytes/sec

      5 minute drop rate, 0 pkts/sec

failover:

        received (in 910613.080 secs):

                32391240 packets        3996588136 bytes

                2 pkts/sec      4002 bytes/sec

        transmitted (in 910613.080 secs):

                32392787 packets        3894177924 bytes

                2 pkts/sec      4002 bytes/sec

      1 minute input rate 1 pkts/sec,  175 bytes/sec

      1 minute output rate 1 pkts/sec,  181 bytes/sec

      1 minute drop rate, 0 pkts/sec

      5 minute input rate 1 pkts/sec,  173 bytes/sec

      5 minute output rate 1 pkts/sec,  180 bytes/sec

      5 minute drop rate, 0 pkts/sec

state:

        received (in 910613.650 secs):

                3193203240 packets      3603824692636 bytes

                3001 pkts/sec   3957002 bytes/sec

        transmitted (in 910613.650 secs):

                574974475 packets       638768128578 bytes

                4 pkts/sec      701003 bytes/sec

      1 minute input rate 1 pkts/sec,  94 bytes/sec

      1 minute output rate 215 pkts/sec,  248131 bytes/sec

      1 minute drop rate, 0 pkts/sec

      5 minute input rate 1 pkts/sec,  94 bytes/sec

      5 minute output rate 220 pkts/sec,  253192 bytes/sec

      5 minute drop rate, 0 pkts/sec

----------------------------------------

Aggregated Traffic on Physical Interface

----------------------------------------

GigabitEthernet0/0:

        received (in 910613.690 secs):

                65108619282 packets     13048153083222 bytes

                71004 pkts/sec  14328004 bytes/sec

        transmitted (in 910613.690 secs):

                61015502631 packets     12223607461115 bytes

                67000 pkts/sec  13423001 bytes/sec

      1 minute input rate 29489 pkts/sec,  6117235 bytes/sec

      1 minute output rate 27129 pkts/sec,  5735537 bytes/sec

      1 minute drop rate, 0 pkts/sec

      5 minute input rate 28705 pkts/sec,  5736986 bytes/sec

      5 minute output rate 26615 pkts/sec,  5582856 bytes/sec

      5 minute drop rate, 0 pkts/sec

GigabitEthernet0/1:

        received (in 910614.410 secs):

                62573130559 packets     12391877851294 bytes

                68003 pkts/sec  13608000 bytes/sec

        transmitted (in 910614.410 secs):

                66607054746 packets     13200515811398 bytes

                73003 pkts/sec  14496000 bytes/sec

      1 minute input rate 27898 pkts/sec,  5880773 bytes/sec

      1 minute output rate 30251 pkts/sec,  6260405 bytes/sec

      1 minute drop rate, 0 pkts/sec

      5 minute input rate 27342 pkts/sec,  5711778 bytes/sec

      5 minute output rate 29427 pkts/sec,  5865616 bytes/sec

      5 minute drop rate, 0 pkts/sec

GigabitEthernet0/2:

        received (in 910614.440 secs):

                3225594482 packets      3742001093928 bytes

                3004 pkts/sec   4109003 bytes/sec

        transmitted (in 910614.440 secs):

                607367417 packets       656024704048 bytes

                1 pkts/sec      720000 bytes/sec

      1 minute input rate 2 pkts/sec,  327 bytes/sec

      1 minute output rate 217 pkts/sec,  253096 bytes/sec

      1 minute drop rate, 0 pkts/sec

      5 minute input rate 2 pkts/sec,  325 bytes/sec

      5 minute output rate 221 pkts/sec,  258249 bytes/sec

      5 minute drop rate, 0 pkts/sec

GigabitEthernet0/3:

        received (in 910615.120 secs):

                0 packets       0 bytes

                0 pkts/sec      0 bytes/sec

        transmitted (in 910615.120 secs):

                0 packets       0 bytes

                0 pkts/sec      0 bytes/sec

      1 minute input rate 0 pkts/sec,  0 bytes/sec

      1 minute output rate 0 pkts/sec,  0 bytes/sec

      1 minute drop rate, 0 pkts/sec

      5 minute input rate 0 pkts/sec,  0 bytes/sec

      5 minute output rate 0 pkts/sec,  0 bytes/sec

      5 minute drop rate, 0 pkts/sec

Internal-Control0/0:

        received (in 910615.140 secs):

                167447499 packets       17309377069 bytes

                4 pkts/sec      19003 bytes/sec

        transmitted (in 910615.140 secs):

                172892596 packets       12199541919 bytes

                1 pkts/sec      13000 bytes/sec

      1 minute input rate 6 pkts/sec,  442 bytes/sec

      1 minute output rate 6 pkts/sec,  458 bytes/sec

      1 minute drop rate, 0 pkts/sec

      5 minute input rate 6 pkts/sec,  442 bytes/sec

      5 minute output rate 6 pkts/sec,  458 bytes/sec

      5 minute drop rate, 0 pkts/sec

Internal-Data0/0:

        received (in 910615.920 secs):

                11815379462 packets     2089588168775 bytes

                12003 pkts/sec  2294004 bytes/sec

        transmitted (in 910615.920 secs):

                11819744701 packets     2091293855897 bytes

                12003 pkts/sec  2296000 bytes/sec

      1 minute input rate 3 pkts/sec,  304 bytes/sec

      1 minute output rate 3 pkts/sec,  304 bytes/sec

      1 minute drop rate, 0 pkts/sec

      5 minute input rate 3 pkts/sec,  305 bytes/sec

      5 minute output rate 3 pkts/sec,  305 bytes/sec

      5 minute drop rate, 0 pkts/sec

Management0/0:

        received (in 910616.490 secs):

                150421 packets  9032160 bytes

                0 pkts/sec      0 bytes/sec

        transmitted (in 910616.490 secs):

                398432 packets  25792376 bytes

                0 pkts/sec      0 bytes/sec

      1 minute input rate 0 pkts/sec,  0 bytes/sec

      1 minute output rate 0 pkts/sec,  0 bytes/sec

      1 minute drop rate, 0 pkts/sec

      5 minute input rate 0 pkts/sec,  0 bytes/sec

      5 minute output rate 0 pkts/sec,  0 bytes/sec

      5 minute drop rate, 0 pkts/sec

i-ASA-5520# show traffic

outside:

        received (in 910668.180 secs):

                65110151423 packets     11643963237805 bytes

                71001 pkts/sec  12786001 bytes/sec

        transmitted (in 910668.180 secs):

                61017722281 packets     11013188883641 bytes

                67003 pkts/sec  12093002 bytes/sec

      1 minute input rate 29171 pkts/sec,  5349689 bytes/sec

      1 minute output rate 26937 pkts/sec,  5178833 bytes/sec

      1 minute drop rate, 3 pkts/sec

      5 minute input rate 28705 pkts/sec,  5119003 bytes/sec

      5 minute output rate 26615 pkts/sec,  5056277 bytes/sec

      5 minute drop rate, 3 pkts/sec

inside:

        received (in 910668.200 secs):

                62574584907 packets     11145910451367 bytes

                68000 pkts/sec  12239002 bytes/sec

        transmitted (in 910668.200 secs):

                66608622302 packets     11762850548667 bytes

                73001 pkts/sec  12916003 bytes/sec

      1 minute input rate 27686 pkts/sec,  5302125 bytes/sec

      1 minute output rate 29918 pkts/sec,  5473008 bytes/sec

      1 minute drop rate, 1 pkts/sec

      5 minute input rate 27341 pkts/sec,  5170050 bytes/sec

      5 minute output rate 29427 pkts/sec,  5232482 bytes/sec

      5 minute drop rate, 1 pkts/sec

management:

        received (in 910668.760 secs):

                150536 packets  6924656 bytes

                0 pkts/sec      2 bytes/sec

        transmitted (in 910668.760 secs):

                398901 packets  11685908 bytes

                0 pkts/sec      3 bytes/sec

      1 minute input rate 0 pkts/sec,  0 bytes/sec

      1 minute output rate 0 pkts/sec,  0 bytes/sec

      1 minute drop rate, 0 pkts/sec

      5 minute input rate 0 pkts/sec,  0 bytes/sec

      5 minute output rate 0 pkts/sec,  0 bytes/sec

      5 minute drop rate, 0 pkts/sec

failover:

        received (in 910668.790 secs):

                32391320 packets        3996597688 bytes

                2 pkts/sec      4001 bytes/sec

        transmitted (in 910668.790 secs):

                32392868 packets        3894187984 bytes

                2 pkts/sec      4002 bytes/sec

      1 minute input rate 1 pkts/sec,  171 bytes/sec

      1 minute output rate 1 pkts/sec,  177 bytes/sec

      1 minute drop rate, 0 pkts/sec

      5 minute input rate 1 pkts/sec,  173 bytes/sec

      5 minute output rate 1 pkts/sec,  180 bytes/sec

      5 minute drop rate, 0 pkts/sec

state:

        received (in 910669.060 secs):

                3193203303 packets      3603824697872 bytes

                3001 pkts/sec   3957002 bytes/sec

        transmitted (in 910669.060 secs):

                574986464 packets       638781929260 bytes

                4 pkts/sec      701003 bytes/sec

      1 minute input rate 1 pkts/sec,  94 bytes/sec

      1 minute output rate 216 pkts/sec,  249164 bytes/sec

      1 minute drop rate, 0 pkts/sec

      5 minute input rate 1 pkts/sec,  94 bytes/sec

      5 minute output rate 220 pkts/sec,  253192 bytes/sec

      5 minute drop rate, 0 pkts/sec

----------------------------------------

Aggregated Traffic on Physical Interface

----------------------------------------

GigabitEthernet0/0:

        received (in 910669.100 secs):

                65110178740 packets     13048456585783 bytes

                71001 pkts/sec  14328003 bytes/sec

        transmitted (in 910669.100 secs):

                61016960052 packets     12223914561682 bytes

                67002 pkts/sec  13423003 bytes/sec

      1 minute input rate 29171 pkts/sec,  5976804 bytes/sec

      1 minute output rate 26937 pkts/sec,  5711992 bytes/sec

      1 minute drop rate, 0 pkts/sec

      5 minute input rate 28705 pkts/sec,  5736986 bytes/sec

      5 minute output rate 26615 pkts/sec,  5582856 bytes/sec

      5 minute drop rate, 0 pkts/sec

GigabitEthernet0/1:

        received (in 910669.400 secs):

                62574616344 packets     12392189039851 bytes

                68000 pkts/sec  13607002 bytes/sec

        transmitted (in 910669.400 secs):

                66608640394 packets     13200823829441 bytes

                73001 pkts/sec  14495000 bytes/sec

      1 minute input rate 27686 pkts/sec,  5850948 bytes/sec

      1 minute output rate 29918 pkts/sec,  6115813 bytes/sec

      1 minute drop rate, 0 pkts/sec

      5 minute input rate 27342 pkts/sec,  5711778 bytes/sec

      5 minute output rate 29427 pkts/sec,  5865616 bytes/sec

      5 minute drop rate, 0 pkts/sec

GigabitEthernet0/2:

        received (in 910669.430 secs):

                3225594624 packets      3742001111764 bytes

                3004 pkts/sec   4109000 bytes/sec

        transmitted (in 910669.430 secs):

                607379397 packets       656038674696 bytes

                1 pkts/sec      720000 bytes/sec

      1 minute input rate 2 pkts/sec,  322 bytes/sec

      1 minute output rate 217 pkts/sec,  254137 bytes/sec

      1 minute drop rate, 0 pkts/sec

      5 minute input rate 2 pkts/sec,  325 bytes/sec

      5 minute output rate 221 pkts/sec,  258249 bytes/sec

      5 minute drop rate, 0 pkts/sec

GigabitEthernet0/3:

        received (in 910670.120 secs):

                0 packets       0 bytes

                0 pkts/sec      0 bytes/sec

        transmitted (in 910670.120 secs):

                0 packets       0 bytes

                0 pkts/sec      0 bytes/sec

      1 minute input rate 0 pkts/sec,  0 bytes/sec

      1 minute output rate 0 pkts/sec,  0 bytes/sec

      1 minute drop rate, 0 pkts/sec

      5 minute input rate 0 pkts/sec,  0 bytes/sec

      5 minute output rate 0 pkts/sec,  0 bytes/sec

      5 minute drop rate, 0 pkts/sec

Internal-Control0/0:

        received (in 910670.140 secs):

                167447845 packets       17309401569 bytes

                4 pkts/sec      19002 bytes/sec

        transmitted (in 910670.140 secs):

                172892953 packets       12199567105 bytes

                1 pkts/sec      13000 bytes/sec

      1 minute input rate 6 pkts/sec,  441 bytes/sec

      1 minute output rate 6 pkts/sec,  457 bytes/sec

      1 minute drop rate, 0 pkts/sec

      5 minute input rate 6 pkts/sec,  442 bytes/sec

      5 minute output rate 6 pkts/sec,  458 bytes/sec

      5 minute drop rate, 0 pkts/sec

Internal-Data0/0:

        received (in 910670.790 secs):

                11815379667 packets     2089588185585 bytes

                12002 pkts/sec  2294002 bytes/sec

        transmitted (in 910670.790 secs):

                11819744906 packets     2091293872707 bytes

                12002 pkts/sec  2296003 bytes/sec

      1 minute input rate 3 pkts/sec,  306 bytes/sec

      1 minute output rate 3 pkts/sec,  306 bytes/sec

      1 minute drop rate, 0 pkts/sec

      5 minute input rate 3 pkts/sec,  305 bytes/sec

      5 minute output rate 3 pkts/sec,  305 bytes/sec

      5 minute drop rate, 0 pkts/sec

Management0/0:

        received (in 910671.460 secs):

                150421 packets  9032160 bytes

                0 pkts/sec      0 bytes/sec

        transmitted (in 910671.460 secs):

                398432 packets  25792376 bytes

                0 pkts/sec      0 bytes/sec

      1 minute input rate 0 pkts/sec,  0 bytes/sec

      1 minute output rate 0 pkts/sec,  0 bytes/sec

      1 minute drop rate, 0 pkts/sec

      5 minute input rate 0 pkts/sec,  0 bytes/sec

      5 minute output rate 0 pkts/sec,  0 bytes/sec

      5 minute drop rate, 0 pkts/sec

i-ASA-5520# sh local-host connection tcp 1000 UDP 1000

Interface state: 1 active, 1 maximum active, 0 denied

Interface management: 0 active, 2 maximum active, 0 denied

Interface inside: 338 active, 968 maximum active, 0 denied

Interface outside: 2456 active, 5294 maximum active, 0 denied

Interface failover: 1 active, 1 maximum active, 0 denied

i-ASA-5520# show local-host connection udp 2000 tcp 2000

Interface state: 1 active, 1 maximum active, 0 denied

Interface management: 0 active, 2 maximum active, 0 denied

Interface inside: 341 active, 968 maximum active, 0 denied

Interface outside: 2450 active, 5294 maximum active, 0 denied

Interface failover: 1 active, 1 maximum active, 0 denied

i-ASA-5520#

Julio Carvajal Thu, 05/23/2013 - 10:13

Hello Sergey,

Wow, huge amount of errors on the interfaces

Add the following to clear the counters

clear interface

Then after 5 minutes

show interface | include errors

Then

sh local-host all connections tcp 1000 udp 1000

show local-host all connections tcp 2000 udp 2000

Based on the previous outputs we can determine that the ASA is not able to handle the traffic being sent on the interfaces, right now I want to determine what is the traffic being sent.

Regards

Mokeev.tel Thu, 05/23/2013 - 10:34

i-ASA-5520# sh clock

14:19:12.569 EDT Thu May 23 2013

i-ASA-5520# show interface | include errors

        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

        0 output errors, 0 collisions, 0 interface resets

        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

        0 output errors, 0 collisions, 0 interface resets

        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

        0 output errors, 0 collisions, 0 interface resets

        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

        0 output errors, 0 collisions, 0 interface resets

        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

        0 output errors, 0 collisions, 0 interface resets

i-ASA-5520# sh clock

14:24:02.429 EDT Thu May 23 2013

i-ASA-5520# show interface | include errors

        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

        0 output errors, 0 collisions, 0 interface resets

        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

        0 output errors, 0 collisions, 0 interface resets

        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

        0 output errors, 0 collisions, 0 interface resets

        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

        0 output errors, 0 collisions, 0 interface resets

        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

        0 output errors, 0 collisions, 0 interface resets

i-ASA-5520#

i-ASA-5520# sh local-host connection tcp 1000 UDP 1000

Interface state: 1 active, 1 maximum active, 0 denied

Interface management: 0 active, 2 maximum active, 0 denied

Interface inside: 345 active, 968 maximum active, 0 denied

Interface outside: 2370 active, 5294 maximum active, 0 denied

Interface failover: 1 active, 1 maximum active, 0 denied

i-ASA-5520# show local-host connection udp 2000 tcp 2000

Interface state: 1 active, 1 maximum active, 0 denied

Interface management: 0 active, 2 maximum active, 0 denied

Interface inside: 347 active, 968 maximum active, 0 denied

Interface outside: 2373 active, 5294 maximum active, 0 denied

Interface failover: 1 active, 1 maximum active, 0 denied

i-ASA-5520#

Julio Carvajal Thu, 05/23/2013 - 10:37

Hello Sergey,

Okey,

I think a capture will show the offending traffic if there is one but before doing that, let me analize some outputs from this discussion

Regards

Mokeev.tel Thu, 05/23/2013 - 11:17

Julio, please, say

1) Is it safe to run capture on such load?

2) If it is Ok, how to run it and diag?

And, Joe wrote about profiling:

cpu profile activate 18000 (wait 3 minutes)
show cpu profile dump
clear cpu profile dump

How can I analyze such dumps?

I think such dump help me to diag, what process and how much resources it uses. Correct?

Julio Carvajal Thu, 05/23/2013 - 11:47

1) Is it safe to run capture on such load?

If you want to fix or determine what the issue is : Yes, but it must be done carefully

2) If it is Ok, how to run it and diag?

http://www.cisco.com/en/US/products/ps6120/products_tech_note09186a0080a9edd6.shtml

That's the link to create capture, you will first create one on the inside interface, matching all traffic, let it run for a few seconds and then check it,

Note: Use the headers-only option for the capture

Regards

Mokeev.tel Thu, 05/23/2013 - 21:41

So, I need to run

access-list asdm_cap_selector_inside extended permit ip any any

capture capin headers-only interface inside access-list asdm_cap_selector_inside

and after few seconds

no capture capin headers-only interface inside access-list asdm_cap_selector_inside

Correct?

And after

show capture capin

How many second use for running?

I afraid to lost connection.

Julio Carvajal Thu, 05/23/2013 - 21:56

Hello,

Before removing the capture you will need to see what you got and make sure you see a relevant pattern,

If you remove it first then you will lost the captured data.

How many seconds... Well if your network is still on 60 % it means that we should be able to see the traffic withing a few seconds (5-15)

Remember(you must have a baseline of the CPU usage on all of your network devices for a better approach on this cases)

You could even donwload the capture to your Desktop so you could analize it more deeply

Regards,

Mokeev.tel Thu, 05/23/2013 - 22:29

At my sample, is it right for ALL trafic?

I'll run it on weekend. I don't have such experience and I avoiding to break something.

So, what at captured data needs my attention? And what other captures to run?

If ASAs run at stack, will heavy capture break both devices? Or does it break one device and switch to other?

Julio Carvajal Fri, 05/24/2013 - 14:32

Hello Sergey,

yeah, it is.

Make sure that over the weekend you have the same CPU, otherwise you will not capture the traffic need it.

The capture must be taken when the ASA has the CPU problems,

What to look for in the capture.. Basically any relevant amount of connections per host,

It's going to be just for 1-15 seconds and the CPU is in 60 %. It will not break anything,

Regards,

Julio

Remember to rate all of the helpful posts Sergey

Mokeev.tel Fri, 05/24/2013 - 18:24

Ok, I have lower cpu-u, so, I ran

iland-ASA-5520# sh pro cpu-u n

PC         Thread       5Sec     1Min     5Min   Process

081a86c4   6c1afa08    26.9%    26.2%    26.7%   Dispatch Unit

083ccbfc   6c1a17a0     0.1%     0.1%     0.1%   fover_health_monitoring_thread

08bde96c   6c18c8f0     0.1%     0.1%     0.6%   ssh

iland-ASA-5520# capture c headers-only interface inside access-list captraffic

iland-ASA-5520# sh pro cpu-u n

PC         Thread       5Sec     1Min     5Min   Process

081a86c4   6c1afa08    35.3%    27.1%    26.8%   Dispatch Unit

083ccbfc   6c1a17a0     0.1%     0.1%     0.1%   fover_health_monitoring_thread

08bde96c   6c18c8f0     0.1%     0.1%     0.5%   ssh

iland-ASA-5520# sh clo

iland-ASA-5520# sh clock

21:51:25.086 EDT Fri May 24 2013

iland-ASA-5520# sh pro cpu-u n

PC         Thread       5Sec     1Min     5Min   Process

081a86c4   6c1afa08    35.9%    29.4%    27.4%   Dispatch Unit

083ccbfc   6c1a17a0     0.1%     0.1%     0.1%   fover_health_monitoring_thread

08bde96c   6c18c8f0     0.0%     0.1%     0.5%   ssh

iland-ASA-5520#

iland-ASA-5520# sh clock

21:51:47.946 EDT Fri May 24 2013

iland-ASA-5520# no capture c headers-only interface inside access-list captraf$

iland-ASA-5520# sh pro cpu-u n

PC         Thread       5Sec     1Min     5Min   Process

081a86c4   6c1afa08    30.9%    30.0%    27.6%   Dispatch Unit

083ccbfc   6c1a17a0     0.1%     0.1%     0.1%   fover_health_monitoring_thread

08bde96c   6c18c8f0     0.6%     0.2%     0.5%   ssh

iland-ASA-5520#

I have inscreased cpu usage about 8-10%, will it increased such with usage 60-70% on workdays?

Next, I have captured less, then 2 seconds. If it die small buffer, do I need increase it? If yes, how?

iland-ASA-5520# sh capture c

7290 packets captured

   1: 20:49:42.370921 IP.55942 > 10.5.0.86.443: . ack 3426703542 win 555

...

7290: 20:49:43.519809 10.5.1.6.5645 > IP.4728: R 0:0(0) ack 3319581259 win 0

And about results, I see most connections are tcp beetween public IP and private on port 443. I have 4215 of 7290 such connections.  And 2204 connections to app port, 422 to public 80 port.

I have 134 connectins udp from 4500 to 4500 port. I don't know, what is it. And other small lines on app ports.

All connections go throw nat. So may be it is my problem? What can I do next?

Julio Carvajal Fri, 05/24/2013 - 18:30

Hi Sergey,

It will increase the CPU around that %.

To change the buffer size you do it in the capture command,

Again, the CPU is at 26% .. You have now see how the network behaves on that percent,

You then will see the results on Monday, if there is any difference,

Again is hard to troubleshoot this problems via messages as there is nothing like being on the console taking the right outpouts and logs,

Hope you have luck next week

Regards

Mokeev.tel Sun, 05/26/2013 - 02:30

Hi Julio,

I'll increase buffer and I'll check it on business days.

Next, what findings can we get here? I think trafic is legal (no viruses trafic) and apps related.

I think I'll get same picture on next days.

I talk boss, he don't authorize me. So, I have just this way.

And, Julio, can you connect here https://supportforums.cisco.com/message/3946884?

Julio Carvajal Sun, 05/26/2013 - 12:53

Hello Sergey,

Yeah, I know running this kind of captures might affect the performance so managment would no authorize it,

If you get the same results then this is expected( I mean 65% would be the baseline)

Final recommendations:

-Keep an eye on the performance of the ASA so you can determine whether this is expected or not as right now we do not know if this is even a cpu case or not

-Keep checking the interface errors counting to determine if everything is going fine,

Regards

Mokeev.tel Tue, 05/28/2013 - 09:54

Hi, I've run capture on work day, so

i-ASA-5520# sh clock

12:54:57.869 EDT Tue May 28 2013

i-ASA-5520# sh pro cpu-u n

PC         Thread       5Sec     1Min     5Min   Process

081a8999   6c1afa08    79.4%    68.4%    67.6%   Dispatch Unit

083ccbfc   6c1a17a0     0.1%     0.1%     0.1%   fover_health_monitoring_thread

08bde96c   6c18daa8     1.0%     0.2%     2.4%   ssh

i-ASA-5520# sh pro cpu-u n

PC         Thread       5Sec     1Min     5Min   Process

081a86c4   6c1afa08    96.4%    70.7%    68.1%   Dispatch Unit

08bde96c   6c18daa8     0.3%     0.2%     2.3%   ssh

i-ASA-5520# sh pro cpu-u n

PC         Thread       5Sec     1Min     5Min   Process

081a88af   6c1afa08    96.4%    70.7%    68.1%   Dispatch Unit

08bde96c   6c18daa8     0.3%     0.2%     2.3%   ssh

i-ASA-5520# sh pro cpu-u n

PC         Thread       5Sec     1Min     5Min   Process

081a88af   6c1afa08    96.4%    70.7%    68.1%   Dispatch Unit

08bde96c   6c18daa8     0.3%     0.2%     2.3%   ssh

i-ASA-5520# sh cloc

12:55:06.839 EDT Tue May 28 2013

i-ASA-5520# sh cloc

12:55:08.179 EDT Tue May 28 2013

i-ASA-5520# sh cloc

12:55:09.049 EDT Tue May 28 2013

i-ASA-5520# sh cloc

12:55:09.779 EDT Tue May 28 2013

i-ASA-5520# no capture c headers-only interface inside access-list captraf$

i-ASA-5520# sh pro cpu-u n

PC         Thread       5Sec     1Min     5Min   Process

081a86c4   6c1afa08    66.6%    73.8%    69.0%   Dispatch Unit

083ccbfc   6c1a17a0     0.1%     0.1%     0.1%   fover_health_monitoring_thread

08bde96c   6c18daa8     1.2%     0.4%     2.2%   ssh

i-ASA-5520#

I see cpu-u jump to 96%. I got big output. And all lines are beween some private and public IPs. So:

1) What to check at capture?

2) How to copy/redirect capture result via ssh or tftp?

I've captured

475383 packets captured

   1: 12:54:55.319670 10.1.0.2.22 > 10.5.0.14.36003: P 2217744027:2217744091(64) ack 949026555 win 32768

....

to

...

475381: 12:55:03.514484 IP.48838 > 10.5.1.66.443: . ack 1650626495 win 255

475382: 12:55:03.514499 IP.50352 > 10.5.1.66.443: . ack 3701255473 win 253

475383: 12:55:03.514499 IP.21556 > 10.5.1.26.443: . ack 2558795455 win 258

475383 packets shown

zerger.andy Fri, 05/31/2013 - 15:11

Hello Everyone, I had a question about this firewall stays at too high in CPU for its bandwidth - actually it is the same device that Sergey is working on. Hi Sergey

Normally during working hours the total traffic on the asa per second is between 60M/s and 80M/s and spikes to around 120M/s.

To add, the asa stays at high CPU even when the traffic drops over the non-usage hours, currently doing %35-40% or 45%-55% CPU for prolonged spikes with these kinds of numbers on the sh interface stats:

outside:

        5 minute input rate 3206 pkts/sec,  1016432 bytes/sec

and inside:

        5 minute input rate 2938 pkts/sec,  380279 bytes/sec

So in general I think the ~1.5M bytes/sec probably should be causing the current usage level.

ASA-5520# sh conn count

6369 in use, 24426 most used

ASA-5520# sh xlate count

1517 in use, 9204 most used

The one thing I noticed today that drove me to the forums to ask: I found this in the "show processes memory" output, it attempted to print binary and broke my putty scrollback:

ASA-5520# sh processes memory

--------------------------------------------------------------

Allocs   Allocated       Frees         Freed           Process

          (bytes)                      (bytes)

--------------------------------------------------------------

0        0               0             0               *System Main*

2635     18446088        150           22582           Init Thread

0        0               0             0               PTHREAD-559

0        0               0             0               PTHREAD-560

0        0               0             0               PTHREAD-561

50       1834            4             1098            DATAPATH-0-562

1        24              1             24              @äl

1        24              1             24              @äl

1        256             1             256             @äl

1        128             1             128             @äl

1        128             1             128             @äl

1        128             1             128             @äl

1        24              1             24              XêqðêqWw%

1        24              1             24              XêqðêqWw%

1        24              1             24              XêqðêqWw%

1        24              1             24              XêqðêqWw%

1        256             1             256             XêqðêqWw%

1        128             1             128             XêqðêqWw%

1        128             1             128             XêqðêqWw%

1        128             1             128             XêqðêqWw%

1        24              1             24              (çíqïÍ¡ÐÔ

1        24              1             24              (çíqïÍ¡ÐÔ

1        24              1             24              (çíqïÍ¡ÐÔ

1        24              1             24              (çíqïÍ¡ÐÔ

1        256             1             256             (çíqïÍ¡ÐÔ

1        128             1             128             (çíqïÍ¡ÐÔ

1        128             1             128             (çíqïÍ¡ÐÔ

1        128             1             128             (çíqïÍ¡ÐÔ

1        24              1             24              (çíqïÍ¡ÐÔ

1        24              1             24              (çíqïÍ¡ÐÔ

1        24              1             24              (çíqïÍ¡ÐÔ

1        24              1             24              (çíqïÍ¡ÐÔ

1        256             1             256             (çíqïÍ¡ÐÔ

1        128             1             128             H»üq

1        128             1             128             H»üq

1        128             1             128             H»üq

1        24              1             24              (zqX+qWw%

1        24              1             24              (zqX+qWw%

1        24              1             24              (zqX+qWw%

1        24              1             24              (zqX+qWw%

1        256             1             256             (zqX+qWw%

1        128             1             128             (zqX+qWw%

1        128             1             128             (zqX+qWw%

1        128             1             128             (zqX+qWw%

1        24              1             24              [email protected]]

1        24              1             24

1        24              1             24

1        24              1             24

1        256             1             256

no  port-object eq 238ïÍ¡ïÍîóÐÔ        128             object eq 23854

no  port-object eq 238ïÍ¡ïÍîóÐÔ        128             object eq 23854

no  port-object eq 238ïÍ¡ïÍîóÐÔ        128             object eq 23854

1        154             1             154             ÿ¡q

1        128             1             128             `.öq

1        128             1             128             `.öq

1        128             1             128             `.öq

1        60              1             60              `.öq

1        24              1             24              `.öq

1        24              1             24              `.öq

1        24              1             24              `.öq

1        24              1             24              `.öq

1        128             1             128             `.öq

1        128             1             128             `.öq

1        128             1             128             `.öq

1        24              1             24              `.öq

1        128             1             128             HªqÈLrWw%

1        128             1             128             HªqÈLrWw%

1        128             1             128             HªqÈLrWw%

1        60              1             60              HªqÈLrWw%

1        24              1             24              HªqÈLrWw%

1        24              1             24              HªqÈLrWw%

1        24              1             24              HªqÈLrWw%

1        24              1             24              HªqÈLrWw%

1        128             1             128             HªqÈLrWw%

1        128             1             128             HªqÈLrWw%

1        128             1             128             HªqÈLrWw%

1        24              1             24              HªqÈLrWw%

1        128             1             128

1        128             1             128

1        128             1             128

1        60              1             60

1        24              1             24

1        24              1             24

1        24              1             24

1        24              1             24

1        128             1             128

1        128             1             128

1        128             1             128

1        24              1             24

1        24              1             24

1        128             1             128             +

1        128             1             128             +

1        128             1             128             +

1        60              1             60              +

1        24              1             24              +

1        24              1             24              +

1        24              1             24              +

1        24              1             24              +

1        128             1             128             +

1        128             1             128             +

1        128             1             128             +

1        24              1             24              +

1        24              1             24              [email protected]

1        24              1             24              [email protected]

1        24              1             24              [email protected]

1        24              1             24              [email protected]

1        128             1             128             [email protected]

1        128             1             128

1        128             1             128             [email protected]

1        24              1             24              [email protected]

1        256             1             256

1        154             1             154             Ø

                                                        28842751203459245106035   2886301327    3460889994696   Dispatch Unit

4        4208            0             0               RADIUS Proxy Listener

2        32              0             0               CP Server Process

0        0               0             0               pci_nt_bridge

0        0               0             0               RADIUS Proxy Time Keeper

172      201280          68            9000            rpc_server

2        152             0             0               CF OIR

7        14163           0             0               Integrity FW Task

2293     29527808        958           26363524        rtcli async executor process

0        0               0             0               lina_int

0        0               0             0               Chunk Manager

0        0               0             0               TLS Proxy Inspector

1208     1988444         1183          1972000         ci/console

1        5242924         6             16712           PIX Garbage Collector

0        0               0             0               emweb/cifs_timer

113      331602          54            12192           fover_thread

4771     76336           4771          114504          npshim_thread

381      5461            854           34413           IP Address Assign

3        8356            0             0               netfs_mount_handler

62430963 3137838648      56485141      3053178710      lu_ctl

1        24              0             0               EAPoUDP-sock

1        640             0             0               Reload Control Thread

0        0               0             0               QoS Support Module

170      28256           207291        35012280        arp_timer

36       3321396         36            3321396         update_cpu_usage

0        0               0             0               EAPoUDP

28608    7929838         33720         8129610         aaa

0        0               0             0               Client Update Task

0        0               0             0               arp_forward_thread

0        0               0             0               health_check

36       1714117         11            33190           UserFromCert Thread

0        0               32            165466          Checkheaps

0        0               0             0               Lic TMR

6        1667            2             66              Boot Message Proxy Process

0        0               126           1008976         tcp_fast

189      17112           200           1033535         NIC status poll

1258     867452          1180          238937          CMGR Server Process

0        0               29            124388          tcp_slow

0        0               0             0               CMGR Timer Process

132683   3435992         132568        3483204         Quack process

0        0               0             0               udp_timer

373707   3214453         745669        6417041         emweb/https

1055     590800          0             0               Session Manager

0        0               0             0               CTCP Timer process

64092    3219860         67857         4792396         fover_rx

371616   2972928         0             0               Timekeeper

0        0               0             0               L2TP data daemon

21107554010604038498     190973066     10322596778     fover_tx

817649   1135312216      791799        297867595       Unicorn Proxy Thread

3        2591535         0             0               uauth

0        0               0             0               L2TP mgmt daemon

402025   20197088        363736        19660888        fover_tx_2

0        0               0             0               Uauth_Proxy

0        0               19            150304          ppp_timer_thread

173465872387146067593     1569452800    84833061869     fover_ip

1384     140752          288           24696           RIP Router

325297   13424196        289910        12869468        dbgtrace

0        0               6             16712           vpnlb_timer_thread

1177     1970463         1178          1970475         fover_rep

1161563238101303716      116131778     8078250159      snmp

0        0               508           12192           RIP Send

517113   60844046        521241        38778697        IPsec message handler

21168707110760671479     191551335     10468568351     fover_parse

3873253  276114276       343           73328           IKE Receiver

0        0               202           47472           SSL

2211383  609816079       294222        42358370        CTM message handler

1        44              1             44              fover_ifc_test

39       75062           7             112             listen/ssh

0        0               0             0               SMTP

0        0               0             0               NAT security-level reconfiguration

2247728372112921627586    2033658751    109924713734    fover_health_monitoring_thread

0        0               19            150304          557mcfix

1        36              7             16748           Logger

0        0               0             0               ICMP event handler

11424    573920          10352         700636          ha_trans_ctl_tx

0        0               0             0               557statspoll

0        0               0             0                Syslog Retry Thread

0        0               0             0               Dynamic Filter VC Housekeeper

1454613957307706124      131607914     7113795052      ha_trans_data_tx

96       14988           0             0               CP Client Process

0        0               0             0               Thread Logger

0        0               0             0               IP Background

8225     3267873         4927          2522617         fover_FSM_thread

33782225 1151761030      63945473      2550570792      vpnfol_thread_msg

141232   8293068         277025        40510553        tmatch compile thread

122866981961724408638     1084307036    58617450838     lu_rx

1851875028888607962      177832614     8785878805      vpnfol_thread_timer

0        0               0             0               Crypto PKI RECV

168      8440            152           8216            lu_dynamic_sync

44       117004          2             20              vpnfol_thread_sync

0        0               0             0               netfs_vnode_reclaim

0        0               0             0               Crypto CA

0        0               0             0               IP Thread

0        0               27            99780           vpnfol_thread_unsent

0        0               0             0               CERT API

40653779 2042378490      36782116      1988193638      ARP Thread

0        0               0             0               Integrity Fw Timer Thread

0        0               0             0               uauth_urlb clean

32       16412           353           59304           icmp_thread

0        0               8             4032            vPif_stats_cleaner

0        0               16            141971          pm_timer_thread

0        0               2             336             udp_thread

0        0               250           6375            ssh/timer

3        8356            0             0               vpnlb_thread

0        0               3             8356            IKE Timekeeper

0        0               52            832             tcp_thread

2        5664            0             0               block_diag

30       908491          0             0               netfs_thread_init

23577503 5672608039      21559530      5336212605      IKE Daemon

125      14948           14185         191936          SNMP Notify Thread

217068   9108714         194133        8739370         ssh

On-disk firmware md5sum checks out with the expected number from Cisco, and the ASA is not having a problem with free memory, it just occurs to me that something acting screwy in the running firmware could cause this binary-print in 'show processes memory' and potentially be related to other strange problems,

Any opinions or experience on whether or not this binary-print in 'sh processes memory' could be relevant?

Thank you very much for your participation so far.

plotniku7 Fri, 06/07/2013 - 06:41

Hi,

i had the same problem with pair of ASA 5510.

after many days of investigation i saw that in one of my interface vlan, there was an VM generating 40Mbps of multicast traffic that was beeing droped by firewall.

i have stopped the multicast traffic frmo VM and CPU droped with 20 %

Thanks

Actions

This Discussion