High CPU Usage / Dropped Packets - Switch Blade WS-CBS3120X-S

juanmacias.0608 · ‎06-26-2012

Hi all,

I have a couple of Switches Blade 3120, working as active-standby model (HSRP) on a new site deployment. There are other 20 sites more or less, working on the same model, without issues. But in this one, we are seeing a high cpu usage. The traffic going through the platform is 600Mbps (on peaks), and in this case we have 40% of CPU usage. Traffic should be close to 3 Gbps. When we tried to send the whole traffic through the platform, active switch began to drop packets on the majority of interfaces.

When we analyze the CPU usage, there is a special process called "HL3U bkgrd proce" always have the most CPU use, but we do not know what concerns. We do not know if it is caused because there are PBRs configured. It should not matter. How I mentioned, there are other sites working fine and have had always the same PBR number.

Could you guys help us?. Any idea what is causing the high usage?. Is there a special debug we could to perform to diagnose the issue?. Also, we have seen a high interrupt CPU usage (9% in this case).

Find attached the whole diagnosis outputs.

Thanks for your assistance guys.

Cheers,

Juan Pablo

bog-sib-INT-rtr-1#show processes cpu sorted 5sec

CPU utilization for five seconds: 30%/9%; one minute: 25%; five minutes: 23%

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

157 140004809 107071220 1307 14.24% 10.19% 9.01% 0 HL3U bkgrd proce

119 6860957 1519183 4516 0.79% 0.59% 0.53% 0 hpm counter proc

166 2511492 302802 8294 0.15% 0.15% 0.15% 0 HQM Stack Proces

199 4182906 15255882 274 0.15% 0.21% 0.20% 0 IP Input

357 237531 782101 303 0.15% 0.03% 0.00% 0 IP SNMP

186 101 148 682 0.15% 0.09% 0.02% 1 Virtual Exec

242 63071 2330717 27 0.15% 0.02% 0.00% 0 CEF: IPv4 proces

12 163754 620353 263 0.15% 0.01% 0.00% 0 ARP Input

9 0 2 0 0.00% 0.00% 0.00% 0 License Client N

8 41 1827 22 0.00% 0.00% 0.00% 0 WATCH_AFS

11 50 4 12500 0.00% 0.00% 0.00% 0 Image License br

7 0 2 0 0.00% 0.00% 0.00% 0 Timers

bog-sib-INT-rtr-1#sh ip cef summary

IPv4 CEF is enabled for distributed and running

VRF Default

119 prefixes (119/0 fwd/non-fwd)

Table id 0x0

Database epoch: 2 (119 entries at this epoch)

Leo Laohoo · ‎06-26-2012

Haven't played with this blade for a long time ...

1. What IOS version do you have?

2. Can you post the output to the command "sh interface "?

juanmacias.0608 · ‎06-26-2012

Hi Leolaohoo,

I had not played with this one too !!!!...

1). IOS version (It was recently updated)

----------------------------------------------------------

bog-sib-INT-rtr-1#sh ver

Cisco IOS Software, CBS31X0 Software (CBS31X0-UNIVERSALK9-M), Version 12.2(58)SE1, RELEASE SOFTWARE (fc1)

Technical Support: http://www.cisco.com/techsupport

Compiled Thu 05-May-11 04:08 by prod_rel_team

ROM: Bootstrap program is CBS31X0 boot loader

BOOTLDR: CBS31X0 Boot Loader (CBS31X0-HBOOT-M) Version 12.2(0.0.951)SE3, CISCO DEVELOPMENT TEST VERSION

bog-sib-INT-rtr-1 uptime is 2 weeks, 3 days, 17 hours, 14 minutes

System returned to ROM by power-on

System restarted at 00:59:27 UTC Sat Jun 9 2012

System image file is "flash:cbs31x0-universalk9-mz.122-58.SE1.bin"

2). What interface do you want to see?, do you want to see all interfaces? . This switch has 16 interfaces that connect servers, and other going to our client. Below, the state of the two kind of interfaces:

Interface to Client (Bearer)

--------------------------------------

TenGigabitEthernet1/0/1 is up, line protocol is up (connected)

Hardware is Ten Gigabit Ethernet, address is 001f.275d.d81b (bia 001f.275d.d81b)

Description: BearerNContent_Aggregrate

MTU 1500 bytes, BW 10000000 Kbit/sec, DLY 10 usec,

reliability 255/255, txload 10/255, rxload 14/255

Encapsulation ARPA, loopback not set

Keepalive not set

Full-duplex, 10Gb/s, link type is auto, media type is 10GBase-LR

input flow-control is off, output flow-control is unsupported

ARP type: ARPA, ARP Timeout 04:00:00

Last input 00:00:00, output 2w3d, output hang never

Last clearing of "show interface" counters 07:07:56

Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0

Queueing strategy: fifo

Output queue: 0/40 (size/max)

5 minute input rate 562469000 bits/sec, 83641 packets/sec

5 minute output rate 430500000 bits/sec, 73141 packets/sec

2020563158 packets input, 1739897855828 bytes, 0 no buffer

Received 13257 broadcasts (13257 multicasts)

0 runts, 0 giants, 0 throttles

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

0 watchdog, 13257 multicast, 0 pause input

0 input packets with dribble condition detected

1745065310 packets output, 1347244137726 bytes, 0 underruns

0 output errors, 0 collisions, 0 interface resets

0 unknown protocol drops

0 babbles, 0 late collision, 0 deferred

0 lost carrier, 0 no carrier, 0 pause output

0 output buffer failures, 0 output buffers swapped out

Interface to Server

--------------------------

GigabitEthernet1/0/8 is up, line protocol is up (connected)

Hardware is Gigabit Ethernet, address is 001f.275d.d808 (bia 001f.275d.d808)

Description: bog-15

MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,

reliability 255/255, txload 15/255, rxload 12/255

Encapsulation ARPA, loopback not set

Keepalive set (10 sec)

Full-duplex, 1000Mb/s, link type is auto, media type is 1000BaseX

input flow-control is off, output flow-control is unsupported

ARP type: ARPA, ARP Timeout 04:00:00

Last input never, output 00:00:17, output hang never

Last clearing of "show interface" counters 07:09:12

Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 19418

Queueing strategy: fifo

Output queue: 0/40 (size/max)

5 minute input rate 47705000 bits/sec, 7155 packets/sec

5 minute output rate 58897000 bits/sec, 8011 packets/sec

178178750 packets input, 153802177226 bytes, 0 no buffer

Received 4091 broadcasts (0 multicasts)

0 runts, 0 giants, 0 throttles

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

0 watchdog, 0 multicast, 0 pause input

0 input packets with dribble condition detected

212233312 packets output, 206621942776 bytes, 0 underruns

0 output errors, 0 collisions, 0 interface resets

0 unknown protocol drops

0 babbles, 0 late collision, 0 deferred

0 lost carrier, 0 no carrier, 0 pause output

0 output buffer failures, 0 output buffers swapped out

Thanks for your help. I am losing my hair with this issue.

Cheers,

Juan P.

Leo Laohoo · ‎06-27-2012

Total output drops: 19418

Hmmmm ...

What's connected to GigabitEthernet1/0/8? It sounds like the interface is pushing as much as it can but the remote end is can't process the inbound data and the switchport is dropping the packets.