input queue drops

[MARCELO MATURO]

I have an RSM router in a Catalyst 5509 module. There are several vlans and I have a lot of input queue drops in vlans interfaces. I have raised the hold-queue to 200 in each interface, but the problem continues.

Thanks. Regards

[daniel.bowen]

Couple of quick questions regarding this problem:

1. Does the CPU peak near 100% at certain times (when you get these input queue drops)?

2. What is the throttle count on the affected interface(s)?

3. How many users do you have attached to the affected interface(s)?

4. What protocols do you run?

regards

Daniel,

[MARCELO MATURO]

the cpu is always normal, the throttle count is low. There are about 600 ws in that interface. There is running IP, IPX, bridging.

RSM_FLORIDA#sh interfaces vlan 5

Vlan5 is up, line protocol is up

Hardware is Cat5k Virtual Ethernet, address is 0090.ab80.e400 (bia 0090.ab80.e400)

Internet address is 10.2.135.4/21

MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec, rely 255/255, load 2/255

Encapsulation ARPA, loopback not set, keepalive set (10 sec)

ARP type: ARPA, ARP Timeout 04:00:00

Last input 00:00:00, output 00:00:00, output hang never

Last clearing of "show interface" counters 01:53:04

Queueing strategy: fifo

Output queue 0/40, 0 drops; input queue 0/200, 3955 drops

5 minute input rate 1627000 bits/sec, 317 packets/sec

5 minute output rate 887000 bits/sec, 276 packets/sec

3001373 packets input, 1857296466 bytes, 0 no buffer

Received 154303 broadcasts, 0 runts, 0 giants, 34 throttles

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

2094359 packets output, 1001235557 bytes, 0 underruns

0 output errors, 0 collisions, 0 interface resets

0 output buffer failures, 0 output buffers swapped out

Thanks, regards

[daniel.bowen]

Ok, you are definately getting a traffic storm of some kind. Somewhere, maybe not on this router but one connected you will have one or more interfaces that are generating too much traffic. Probably due to a virus or something. A thottle occurs when too much traffic enters an interface than it can queue, so the interface shuts down temporarily until the queue is clear. Upping the hold queue will help the problem but you really need to find out where the traffic is coming from. How often do these throttles occur? What routers do you have other than the RSM?

Daniel,

[MARCELO MATURO]

Thank you very much. Now I am trying to find out where the traffic is coming from. I agree with you that the problem is a traffic storm.

I have 7206 routers too. The RSM version 11.2(13).

Regards

[daniel.bowen]

Have you had any joy with finding where the traffic is coming from?

Daniel,

[MARCELO MATURO]

I am finding a burst of ipx traffic concident with the drops, about 5 Mbps ipx input , the rest of the traffic lower ( IP, ARP, TB, Spanning Tree).

Marcelo

[jeff]

Perhaps users sending huge attachments through Lotus Notes. You may want to

dig a little and see exactly what the users who are experiencing problems are sending/receiving when they have problems. Could be that segment or VLAN is getting overloaded.

[daniel.bowen]

How long has theis problem been occuring? Just before it began, did you change anything in your topology such as upgrade your LAN speed, or install any new routers?

Daniel,

[jeff]

Question: Even though the interface is showing drops, are you getting any complaints from your users? I find that on heavily used interfaces, some drops will occur, but it is an EXTREMELY SMALL percentage of total traffic. NBD.

Also: It seems that the SIZE of your Packets is VERY large. What kind of apps/traffic are you running?? See my Switch data and compare to yours:

Queueing strategy: fifo

Output queue 0/40, 0 drops; input queue 0/300, 1719 drops

5 minute input rate 2942000 bits/sec, 1105 packets/sec

5 minute output rate 7660000 bits/sec, 1283 packets/sec

339258970 packets input, 2132869750 bytes, 0 no buffer

Received 27589788 broadcasts, 0 runts, 0 giants, 0 throttles

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

0 watchdog, 0 multicast

0 input packets with dribble condition detected

392707361 packets output, 3770267502 bytes, 0 underruns(0/0/0)

0 output errors, 0 collisions, 0 interface resets

0 babbles, 0 late collision, 0 deferred

0 lost carrier, 0 no carrier

0 output buffer failures, 0 output buffers swapped out

[MARCELO MATURO]

There are complaints from some users, mainly lotus notes aplication, that reports networks errors. Perhaps the percent seems low, but in a short

period of time is important. I see too that you don't have throttles.

MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec, rely 255/255, load 1/255 Encapsulation ARPA, loopback not set, keepalive set (10 sec) ARP type: ARPA, ARP Timeout 04:00:00 Last input 00:00:00, output 00:00:00, output hang never Last clearing of "show interface" counters 1d07h Queueing strategy: fifo Output queue 0/40, 0 drops; input queue 0/200, 505704 drops 5 minute input rate 449000 bits/sec, 128 packets/sec 5 minute output rate 214000 bits/sec, 142 packets/sec 26207996 packets input, 1598384356 bytes, 0 no buffer Received 2311908 broadcasts, 0 runts, 0 giants, 693 throttles 0 input errors, 0 CRC, 0 frame, 0 overrun, 320 ignored, 0 abort 20882895 packets output, 1474539777 bytes, 0 underruns 0 output errors, 0 collisions, 0 interface resets 0 output buffer failures, 0 output buffers swapped out

[jeff]

That IS alot of drops. What is the speed and type of the Interface that you are showing here? Is there more than one VLAN on it (Port-channel's) or is just one ?

Try increasing the queue to 300 and see what happens. Sounds like you have a bad upstream switch or device feeding errors down.

[MARCELO MATURO]

The interface is a vlan interface in an RSM router in a 5509 chasis.

Marcelo

[daniel.bowen]

IPX traffic bursts are reasonably usual, is the IPX traffic NDS and NCP?

When the throttles occur, that is the IPX input on the show cpu proc display?

Upping the hold queue will not solve this problem, it will only partially mask it as you obviously have a rogue device or a virus that is causing massive traffic bursts. Couple more questions:

1. what IOS code do you have on the RSM?

2. what interfaces do the throttles occur (input queue drops are common, throttles are the sign that the interface is over-subscribed?

Daniel,

[MARCELO MATURO]

Daniel,

The ipx traffic is NDS a NCP and the protocol is ipx rip.

The IOS is Version 11.2(13)P.

The interfaces that the throttles occur are 2 vlan interface over a total of 15 vlan interfaces define in the RSM router.

Marcelo,