ASA 5525-X underrruns after DATAPATH CPU-HOGs

kerstin-534 · ‎06-17-2014

Hi,

one ASA 5525-X experiences regular underrun drops on one interface. The underruns corresponds to a full TX-ring (bold) and CPU-Hogs of the DATAPATH process. Is there any reason for hogs of the datapath process ? When experincing hogs while NO high traffic on the interface is experienced. The 1-minute CPU rises never abover 30 percent.

Jay Johnston, speaker at Cisco Live 2014 San Francisco in Troubleshooting Firewalls, mentioned that the cause can be a lot of subinterfaces. Why ?

Interface GigabitEthernet0/3 "", is up, line protocol is up
Hardware is i82574L rev00, BW 1000 Mbps, DLY 10 usec
        Full-Duplex(Full-duplex), 1000 Mbps(1000 Mbps)
        Input flow control is unsupported, output flow control is off
        Active member of Redundant4
        MAC address 0006.f6e6.4c40, MTU not set
        IP address unassigned
        35877832491 packets input, 35071910655292 bytes, 0 no buffer
        Received 186155630 broadcasts, 0 runts, 0 giants
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
        0 pause input, 0 resume input
        0 L2 decode drops
        36630896751 packets output, 33324052779933 bytes, 2509559 underruns
        0 pause output, 0 resume output
        0 output errors, 0 collisions, 0 interface resets
        0 late collisions, 0 deferred
        0 input reset drops, 0 output reset drops
        input queue (blocks free curr/low): hardware (499/362)
        output queue (blocks free curr/low): hardware (511/0)

Process:      DATAPATH-0-1244, PROC_PC_TOTAL: 21641, MAXHOG: 5, LASTHOG: 2
LASTHOG At:   09:34:52 MEST Jun 17 2014
PC:           0x0000000000000000 (suspend)

Process:      DATAPATH-0-1244, NUMHOG: 20940, MAXHOG: 5, LASTHOG: 2
LASTHOG At:   09:34:52 MEST Jun 17 2014
PC:           0x0000000000000000 (suspend)
Call stack:   0x000000000041a19e 0x000000000041a373 0x000000000069bb7b
              0x00000000013688cf 0x000000000137382d 0x0000000001378e73
              0x00007ffffeccef3a

firewall# show traffic | beg 0/3
GigabitEthernet0/3:
        received (in 2387365.870 secs):
                35922871958 packets     35111721054384 bytes
                15000 pkts/sec 14707000 bytes/sec
        transmitted (in 2387365.870 secs):
                36673214828 packets     33353535736492 bytes
                15001 pkts/sec 13970001 bytes/sec
      1 minute input rate 12210 pkts/sec, 10802057 bytes/sec
      1 minute output rate 11170 pkts/sec, 7453377 bytes/sec
      1 minute drop rate, 0 pkts/sec
      5 minute input rate 13702 pkts/sec, 12221844 bytes/sec
      5 minute output rate 12902 pkts/sec, 9565241 bytes/sec
      5 minute drop rate, 0 pkts/sec

I would be happy for some feedback.

Cheers,

Fritz

nkarthikeyan · ‎06-17-2014

Hi Fritz,

Normally underrun errors causing because of the over subscription / when you have the Qos enabled in your ASA.

If you have Qos enabled for that interface. You c an try to by disabling the Qos on that interface.
Minimize the sub-interfaces created on an interface.
And then you can try to maximize the throughput by distributing the traffic between 2 BUSes. If you have this option.

3 Maximizing Throughput (ASA 5550)

The ASA 5550 has two internal buses providing copper Gigabit Ethernet and fiber Gigabit Ethernet connectivity. For Slot 1 (Bus 1), you can use either the copper ports or the fiber ports. The copper ports are enabled by default.

For maximum throughput, configure the ASA so that traffic is distributed equally between the two buses. Lay out the network so that traffic enters through one bus and exits through the other.

For example, the following figure shows the ASA configured so that traffic from the unsecure network and the secure network is evenly distributed between Bus 0 and Bus 1. Traffic from hosts on the secured network flows through interface 0/0 on Bus 0 to hosts on the unsecured network. Traffic from hosts on the unsecured network flows through interface 1/0 on Bus 1 to hosts on the secured network.

HTH

Regards

Karthik

kerstin-534 · ‎06-23-2014

Hi Karthik,

I have found the root cause. There is very bursty traffic on one CIFS connection coming in from another Gigabit interface, causing oversubscibing the overall throuput of the destination interface from 1 Gig. The communication between subinterfaces from the destination interface itself never results in overruns, but with 2 different physical interfaces overruns are very likely. So to solve the problem faster interfaces would be necessary (my opinion).

best regards,

Fritz