VLAN Input Queue Drops - 6509 Sup 720

Unanswered Question
Apr 12th, 2010

Hi All,

I did a search out there for VLAN Input Queue Drops and came up with a few threads but none that really seemed to answer the problem.  I have a Vlan on my 6509 that is experiencing a high number of Input Queue drops.  The big problem is that I can't catch it while it's happening.  It always seems to happen of course when I'm not looking at the interface.  That's making it next to impossible to find the cause for the issue.  I'm debating increasing the input queue max above 75 but, I'm leary of causing any sort of performance degredation.  I've seen a load of these issues out there where guys have increased the queue up to 2000 and it didn't seem to help.  I'll take any suggestions, see my snapshot below, this is after I cleared the couters earlier this morning, seems like my input counters are rather high for having ip cef and fast switching enabled globally, the only change I can think of that's happened is we turned on port-security on all the distributed switches which sit in the closets amongst our high-rise but, they arn't a part of this VLAN.:


Vlan44 is up, line protocol is up
  Hardware is EtherSVI, address is 0019.07d4.7400 (bia 0019.07d4.7400)
  Description: Example Data VLAN
  Internet address is 149.208.44.15/22
  MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 5/255
  Encapsulation ARPA, loopback not set
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:00, output 00:00:00, output hang never
  Last clearing of "show interface" counters 3d02h
  Input queue: 0/75/9662/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 19648000 bits/sec, 8919 packets/sec
  5 minute output rate 7143000 bits/sec, 2406 packets/sec
  L2 Switched: ucast: 12985585921 pkt, 11068830624049 bytes - mcast: 4524727 pkt
, 487897744 bytes
  L3 in Switched: ucast: 2517029817 pkt, 316096903865 bytes - mcast: 0 pkt, 0 by
tes mcast
  L3 out Switched: ucast: 226054288 pkt, 152701807644 bytes mcast: 0 pkt, 0 byte
s
     2533488572 packets input, 318996427575 bytes, 0 no buffer
     Received 4132914 broadcasts (286009 IP multicast)
     0 runts, 0 giants, 58 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     242790991 packets output, 157082286957 bytes, 0 underruns
     0 output errors, 0 interface resets
     0 output buffer failures, 0 output buffers swapped out
CSW1_6509E#

I have this problem too.
2 votes
  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 5 (1 ratings)
Loading.
allan.thomas Mon, 04/12/2010 - 15:10

Hi,

The output strongly suggests that your traffic  received on input is being processed switched thus causing congestion  simply because there is simply no route-cache entry to determine the  forwarding decision thus has to be queued before it can be processed.

There  are two options for trying determine the traffic or protocol in  question, either debug ip packet which would be very cpu intensive and  therefore caution is warranted when doing this and preferably should  be logged to the buffer.  Under the circumstances I can appreciate that  this is difficult to determine and any debugging should be done out of  hours which could not actually capture the traffic in question.  

A 'show process cpu' should also help identify which input process is hogging the CPU at time the drops occur, and may help given an indication as to what type of traffic is attributed to the drops.

Alternatively  if you run a 'show buffers input-interface  ' This should indicate what specific ip protocol is prevalent in the buffers at any given time, for example at the end of the output you should be able to identify they keyworrd prot:, such as prot: 1.

Regards

Allan.

Hope this helps, pls rate help poss.

rhornberger Tue, 04/13/2010 - 05:29

I'll see what I can catch with show buff input-int vlan 44...  Right now it of course shows nothing because nothing is happening at the moment.  I'll have to work on my scripting skills, maybe I can come up with something that constantly runs the command and reports back with anything different...  Right now all I see is nothing....  HAHA

CSW1_6509E#show buffers input-interface vlan 44

  Header DataArea  Pool Rcnt  Size Link  Enc    Flags          Input      Output

Thanks for the additional non-intrusive suggestion however!

rhornberger Tue, 04/13/2010 - 05:32

Hmmm...  Now I caught some items in the buffer.  I'll have to try and get more details maybe with a

show buffers input-interface vlan 44 dump or likewise...

CSW1_6509E#show buffers input-interface vlan 44

  Header DataArea  Pool Rcnt  Size Link  Enc    Flags          Input      Output

43B83FE0  803F7C4 Small    1    93    7    1      284           Vl44        None
5004F6A8  8060C44 Small    1    64    0    1      200           Vl44        None
50062F28  8073844 Small    1    64    0    1      200           Vl44        None
5013EAC8  81BD664 Mediu    1   243    7    1      280           Vl44        None

CSW1_6509E#show buffers input-interface vlan 44

  Header DataArea  Pool Rcnt  Size Link  Enc    Flags          Input      Output

43CBDDC0  81F3B84 Mediu    1   240    7    1      280           Vl44        None

rhornberger Tue, 04/13/2010 - 05:49

I keep seeing these filling the packet queue with the occasional other service mixed in but, many many more of these than anything else.


Buffer information for Small buffer at 0x500398A8
  data_area 0x804BC44, refcount 1, next 0x0, flags 0x200
  linktype 1 (ARP), enctype 1 (ARPA), encsize 14, rxtype 45
  if_input 0x44DA00F0 (Vlan44), if_output 0x0 (None)
  inputtime 39w2d (elapsed 00:00:43.180)
  outputtime 00:00:00.000 (elapsed never), oqnumber 65535
  datagramstart 0x804BCBA, datagramsize 60, maximum size 308
  mac_start 0x804BCBA, addr_start 0x804BCBA, info_start 0x0
  network_start 0x804BCC8, transport_start 0x804BCDC, caller_pc 0x403E3570


0804BCB0:                       0019 07D47400            ...Tt.
0804BCC0: 00137216 7B680806 00010800 06040002  ..r.{h..........
0804BCD0: 00137216 7B6895D0 2EC60019 07D47400  ..r.{h.P.F...Tt.
0804BCE0: 95D02C0F 00000000 00000000 00000000  .P,.............
0804BCF0: 00000000 000032                      ......2

Buffer information for Small buffer at 0x50059F68
  data_area 0x806AE44, refcount 1, next 0x500398A8, flags 0x200
  linktype 1 (ARP), enctype 1 (ARPA), encsize 14, rxtype 45
  if_input 0x44DA00F0 (Vlan44), if_output 0x0 (None)
  inputtime 39w2d (elapsed 00:00:38.312)
  outputtime 00:00:00.000 (elapsed never), oqnumber 65535
  datagramstart 0x806AEBA, datagramsize 60, maximum size 308
  mac_start 0x806AEBA, addr_start 0x806AEBA, info_start 0x0
  network_start 0x806AEC8, transport_start 0x806AEDC, caller_pc 0x403E3570


0806AEB0:                       FFFF FFFFFFFF            ......
0806AEC0: 0017A477 00100806 00010800 06040001  ..$w............
0806AED0: 0017A477 001095D0 2C4B0000 00000000  ..$w...P,K......
0806AEE0: 95D02F11 00000000 00000000 00000000  .P/.............
0806AEF0: 00000000 000033                      ......3

Actions

This Discussion