08-08-2008 04:00 PM - edited 03-03-2019 11:05 PM
Guys
i posted a while back on an issue we were having where our branches intermittently dropped throughout the day. we thought we resolved the issue by changing a WAN cable however our celebration was shortlived.
The issue
every so often our IP Phones drop and our telenet application Sitcks (for a little more than 1 SEC. what we noticed is that everytime that happened there was a throttle recorded.
We are currently doing a virus scan and working with TAC but im not reaching any closer to resolving the issue.
what we have done so far
1. Secure EIGRP
2. Change WAN Cables
3. Change Router
4. Enable storm control - broadcast 20.00
5. check all switching ports to see if there were any errors
6. monitor CPU - Nothing over 25%
7. Disable Multicast -PIM - for our paging application
8. increase hold time
9. increase input queue
10. adjusted buffers with TAC assistance
no matter what the issue is still there. The throttles may happen every two days or sometimes not at all for a week and then come back again. everytime it happens, network is lost for a few seconds.
im looking for some directions into troubleshooting this issue. Should i simple connect a sniffer to the network and see what happens?
again our configs are not at all complex.
the environment:
HEAD OFFCE TO BRANCH A AND B
HO = 2851-----4 3750
BRANCH A = 2811 - 3750----3550
BRANCH B = 3750
Any pointers would be greatly appreciated.
the carrier also reports no issue and we are not in a position to disagree at this time since we require proof of what is at fault.
thanks again in advance
08-09-2008 02:06 AM
Hello Arvind,
it looks like you have already done what is possible.
Have you increased the input hold-queue on the interface.
Do you see input drops on the interface ?
the Throttles definition is :
Understanding Throttles
The output of the show interface atm command might display a high number of throttles along with input queue drops. Input queue drops occur when a packet is being process switched. The throttles counter increments when a system buffer is available, but the interface already has the maximum number of packets waiting to be processed in the input hold queue
>>The router temporarily disables the interface to give the interface time to catch up and process the already enqueued packets.
>> this is the problem you see
You can troubleshoot throttles by determining the root cause of why a high number of packets are being process switched.
For general interfaces a link for troubleshooting input drops
http://www.cisco.com/en/US/products/hw/routers/ps133/products_tech_note09186a0080094791.shtml#topic3
It provides suggestions on how to find what traffic causes the problem.
Hope to help
Giuseppe
08-09-2008 04:26 AM
You note both WAN routers and L3 switches, and only a L3 switch in Branch B. Where do you see the throttles? Could you further explain the connectivity between devices? Any QoS/CoS active on any of the devices?
08-09-2008 03:29 PM
thanks for the responses guys
first off we raised the input queue to 120 but that didnt resolve the issue. we also are seeing drops, looks like for every throttle there are about 3 drops...not really consistent as another might be 5 drops etc. - ive read the throttle docs a few times and i understand whats going on but im stumpped as to how to resolve. I also noted that one document stated that Explorer Storms normally causes this type of issue
As for the second reply, the throttles are seen at one of the branches -BRANCH A only.
the baisc architecture is as follows
we have a fibre (ISP) connecting all locations. Head office is a 2851 which is the Hub. BRANCH A is a 2811 and BRANCH B is the L3 Switch.
QOS is enabled -AUTO QOS on all switches and routers. However, QOS on the routers were disabled temporarily while we troubleshoot the issue.
Links are 3MB to each location
Thanks for the responses
PS just logged into the router
SH INT
Internet address is ***********
MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 100Mb/s, media type is T
output flow-control is XON, input flow-control is XON
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters 3d09h
Input queue: 0/75/69/0 (size/max/drops/flushes); Total output drops:
Queueing strategy: fifo
Output queue: 0/40 (size/max)
30 second input rate 4000 bits/sec, 6 packets/sec
30 second output rate 11000 bits/sec, 7 packets/sec
18150098 packets input, 2353919361 bytes, 0 no buffer
Received 140549 broadcasts, 0 runts, 0 giants, 3 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 0 multicast, 0 pause input
0 input packets with dribble condition detected
18012328 packets output, 494258109 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out
SH INT GIG 0/1 SWITCHING
GigabitEthernet0/1 Columbus Communications link to San Fernando$ETH-WAN$
Throttle count 7
Drops RP 104 SP 0
SPD Flushes Fast 0 SSE 0
SPD Aggress Fast 0
SPD Priority Inputs 432623 Drops 0
08-09-2008 04:10 PM
This is a replacement router that we have installed with the defaults so you would not see the input queue adjusted. Cisco asked us if we had another router so we placed this one in the network with the same configuration before the issue happened.
Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: