VLAN Input Queue Drops - 6509 Sup 720

rhornberger · ‎04-12-2010

Hi All,

I did a search out there for VLAN Input Queue Drops and came up with a few threads but none that really seemed to answer the problem. I have a Vlan on my 6509 that is experiencing a high number of Input Queue drops. The big problem is that I can't catch it while it's happening. It always seems to happen of course when I'm not looking at the interface. That's making it next to impossible to find the cause for the issue. I'm debating increasing the input queue max above 75 but, I'm leary of causing any sort of performance degredation. I've seen a load of these issues out there where guys have increased the queue up to 2000 and it didn't seem to help. I'll take any suggestions, see my snapshot below, this is after I cleared the couters earlier this morning, seems like my input counters are rather high for having ip cef and fast switching enabled globally, the only change I can think of that's happened is we turned on port-security on all the distributed switches which sit in the closets amongst our high-rise but, they arn't a part of this VLAN.:

Vlan44 is up, line protocol is up
Hardware is EtherSVI, address is 0019.07d4.7400 (bia 0019.07d4.7400)
Description: Example Data VLAN
Internet address is 149.208.44.15/22
MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 5/255
Encapsulation ARPA, loopback not set
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters 3d02h
Input queue: 0/75/9662/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 19648000 bits/sec, 8919 packets/sec
5 minute output rate 7143000 bits/sec, 2406 packets/sec
L2 Switched: ucast: 12985585921 pkt, 11068830624049 bytes - mcast: 4524727 pkt
, 487897744 bytes
L3 in Switched: ucast: 2517029817 pkt, 316096903865 bytes - mcast: 0 pkt, 0 by
tes mcast
L3 out Switched: ucast: 226054288 pkt, 152701807644 bytes mcast: 0 pkt, 0 byte
s
     2533488572 packets input, 318996427575 bytes, 0 no buffer
     Received 4132914 broadcasts (286009 IP multicast)
     0 runts, 0 giants, 58 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     242790991 packets output, 157082286957 bytes, 0 underruns
     0 output errors, 0 interface resets
     0 output buffer failures, 0 output buffers swapped out
CSW1_6509E#

allan.thomas · ‎04-12-2010

Hi,

The output strongly suggests that your traffic received on input is being processed switched thus causing congestion simply because there is simply no route-cache entry to determine the forwarding decision thus has to be queued before it can be processed.

There are two options for trying determine the traffic or protocol in question, either debug ip packet which would be very cpu intensive and therefore caution is warranted when doing this and preferably should be logged to the buffer. Under the circumstances I can appreciate that this is difficult to determine and any debugging should be done out of hours which could not actually capture the traffic in question.

A 'show process cpu' should also help identify which input process is hogging the CPU at time the drops occur, and may help given an indication as to what type of traffic is attributed to the drops.

Alternatively if you run a 'show buffers input-interface ' This should indicate what specific ip protocol is prevalent in the buffers at any given time, for example at the end of the output you should be able to identify they keyworrd prot:, such as prot: 1.

Regards

Allan.

Hope this helps, pls rate help poss.

rhornberger · ‎04-13-2010

I'll see what I can catch with show buff input-int vlan 44... Right now it of course shows nothing because nothing is happening at the moment. I'll have to work on my scripting skills, maybe I can come up with something that constantly runs the command and reports back with anything different... Right now all I see is nothing.... HAHA

CSW1_6509E#show buffers input-interface vlan 44

Header DataArea Pool Rcnt Size Link Enc Flags Input Output

Thanks for the additional non-intrusive suggestion however!

rhornberger · ‎04-13-2010

Hmmm... Now I caught some items in the buffer. I'll have to try and get more details maybe with a

show buffers input-interface vlan 44 dump or likewise...

CSW1_6509E#show buffers input-interface vlan 44

Header DataArea Pool Rcnt Size Link Enc Flags Input Output

43B83FE0 803F7C4 Small    1    93    7    1      284           Vl44        None
5004F6A8 8060C44 Small    1    64    0    1      200           Vl44        None
50062F28 8073844 Small    1    64    0    1      200           Vl44        None
5013EAC8 81BD664 Mediu    1   243    7    1      280           Vl44        None

CSW1_6509E#show buffers input-interface vlan 44

Header DataArea Pool Rcnt Size Link Enc Flags Input Output

43CBDDC0 81F3B84 Mediu 1 240 7 1 280 Vl44 None

rhornberger · ‎04-13-2010

I keep seeing these filling the packet queue with the occasional other service mixed in but, many many more of these than anything else.

Buffer information for Small buffer at 0x500398A8
data_area 0x804BC44, refcount 1, next 0x0, flags 0x200
linktype 1 (ARP), enctype 1 (ARPA), encsize 14, rxtype 45
if_input 0x44DA00F0 (Vlan44), if_output 0x0 (None)
inputtime 39w2d (elapsed 00:00:43.180)
outputtime 00:00:00.000 (elapsed never), oqnumber 65535
datagramstart 0x804BCBA, datagramsize 60, maximum size 308
mac_start 0x804BCBA, addr_start 0x804BCBA, info_start 0x0
network_start 0x804BCC8, transport_start 0x804BCDC, caller_pc 0x403E3570

0804BCB0: 0019 07D47400 ...Tt.
0804BCC0: 00137216 7B680806 00010800 06040002 ..r.{h..........
0804BCD0: 00137216 7B6895D0 2EC60019 07D47400 ..r.{h.P.F...Tt.
0804BCE0: 95D02C0F 00000000 00000000 00000000 .P,.............
0804BCF0: 00000000 000032 ......2

Buffer information for Small buffer at 0x50059F68
data_area 0x806AE44, refcount 1, next 0x500398A8, flags 0x200
linktype 1 (ARP), enctype 1 (ARPA), encsize 14, rxtype 45
if_input 0x44DA00F0 (Vlan44), if_output 0x0 (None)
inputtime 39w2d (elapsed 00:00:38.312)
outputtime 00:00:00.000 (elapsed never), oqnumber 65535
datagramstart 0x806AEBA, datagramsize 60, maximum size 308
mac_start 0x806AEBA, addr_start 0x806AEBA, info_start 0x0
network_start 0x806AEC8, transport_start 0x806AEDC, caller_pc 0x403E3570

0806AEB0: FFFF FFFFFFFF ......
0806AEC0: 0017A477 00100806 00010800 06040001 ..$w............
0806AED0: 0017A477 001095D0 2C4B0000 00000000 ..$w...P,K......
0806AEE0: 95D02F11 00000000 00000000 00000000 .P/.............
0806AEF0: 00000000 000033 ......3