Router Throttles

a.gooding · ‎08-08-2008

Guys

i posted a while back on an issue we were having where our branches intermittently dropped throughout the day. we thought we resolved the issue by changing a WAN cable however our celebration was shortlived.

The issue

every so often our IP Phones drop and our telenet application Sitcks (for a little more than 1 SEC. what we noticed is that everytime that happened there was a throttle recorded.

We are currently doing a virus scan and working with TAC but im not reaching any closer to resolving the issue.

what we have done so far

1. Secure EIGRP

2. Change WAN Cables

3. Change Router

4. Enable storm control - broadcast 20.00

5. check all switching ports to see if there were any errors

6. monitor CPU - Nothing over 25%

7. Disable Multicast -PIM - for our paging application

8. increase hold time

9. increase input queue

10. adjusted buffers with TAC assistance

no matter what the issue is still there. The throttles may happen every two days or sometimes not at all for a week and then come back again. everytime it happens, network is lost for a few seconds.

im looking for some directions into troubleshooting this issue. Should i simple connect a sniffer to the network and see what happens?

again our configs are not at all complex.

the environment:

HEAD OFFCE TO BRANCH A AND B

HO = 2851-----4 3750

BRANCH A = 2811 - 3750----3550

BRANCH B = 3750

Any pointers would be greatly appreciated.

the carrier also reports no issue and we are not in a position to disagree at this time since we require proof of what is at fault.

thanks again in advance

Giuseppe Larosa · ‎08-09-2008

Hello Arvind,

it looks like you have already done what is possible.

Have you increased the input hold-queue on the interface.

Do you see input drops on the interface ?

the Throttles definition is :

Understanding Throttles

The output of the show interface atm command might display a high number of throttles along with input queue drops. Input queue drops occur when a packet is being process switched. The throttles counter increments when a system buffer is available, but the interface already has the maximum number of packets waiting to be processed in the input hold queue

>>The router temporarily disables the interface to give the interface time to catch up and process the already enqueued packets.

>> this is the problem you see

You can troubleshoot throttles by determining the root cause of why a high number of packets are being process switched.

For general interfaces a link for troubleshooting input drops

http://www.cisco.com/en/US/products/hw/routers/ps133/products_tech_note09186a0080094791.shtml#topic3

It provides suggestions on how to find what traffic causes the problem.

Hope to help

Giuseppe

Joseph W. Doherty · ‎08-09-2008

You note both WAN routers and L3 switches, and only a L3 switch in Branch B. Where do you see the throttles? Could you further explain the connectivity between devices? Any QoS/CoS active on any of the devices?

a.gooding · ‎08-09-2008

thanks for the responses guys

first off we raised the input queue to 120 but that didnt resolve the issue. we also are seeing drops, looks like for every throttle there are about 3 drops...not really consistent as another might be 5 drops etc. - ive read the throttle docs a few times and i understand whats going on but im stumpped as to how to resolve. I also noted that one document stated that Explorer Storms normally causes this type of issue

As for the second reply, the throttles are seen at one of the branches -BRANCH A only.

the baisc architecture is as follows

we have a fibre (ISP) connecting all locations. Head office is a 2851 which is the Hub. BRANCH A is a 2811 and BRANCH B is the L3 Switch.

QOS is enabled -AUTO QOS on all switches and routers. However, QOS on the routers were disabled temporarily while we troubleshoot the issue.

Links are 3MB to each location

Thanks for the responses

PS just logged into the router

SH INT

Internet address is ***********

MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,

reliability 255/255, txload 1/255, rxload 1/255

Encapsulation ARPA, loopback not set

Keepalive set (10 sec)

Full-duplex, 100Mb/s, media type is T

output flow-control is XON, input flow-control is XON

ARP type: ARPA, ARP Timeout 04:00:00

Last input 00:00:00, output 00:00:00, output hang never

Last clearing of "show interface" counters 3d09h

Input queue: 0/75/69/0 (size/max/drops/flushes); Total output drops:

Queueing strategy: fifo

Output queue: 0/40 (size/max)

30 second input rate 4000 bits/sec, 6 packets/sec

30 second output rate 11000 bits/sec, 7 packets/sec

18150098 packets input, 2353919361 bytes, 0 no buffer

Received 140549 broadcasts, 0 runts, 0 giants, 3 throttles

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

0 watchdog, 0 multicast, 0 pause input

0 input packets with dribble condition detected

18012328 packets output, 494258109 bytes, 0 underruns

0 output errors, 0 collisions, 0 interface resets

0 babbles, 0 late collision, 0 deferred

0 lost carrier, 0 no carrier, 0 pause output

0 output buffer failures, 0 output buffers swapped out

SH INT GIG 0/1 SWITCHING

GigabitEthernet0/1 Columbus Communications link to San Fernando$ETH-WAN$

Throttle count 7

Drops RP 104 SP 0

SPD Flushes Fast 0 SSE 0

SPD Aggress Fast 0

SPD Priority Inputs 432623 Drops 0

a.gooding · ‎08-09-2008

This is a replacement router that we have installed with the defaults so you would not see the input queue adjusted. Cisco asked us if we had another router so we placed this one in the network with the same configuration before the issue happened.