input errors on 1841 LAN interface

tato386 · ‎11-05-2011

I use an 1841 router as an internet facing firewall with a 10MB MetroE connection. Lately users started reporting slow internet download speeds and web pages timing out. Bandwidth reports do not show the link as being saturated so I looked at the interfaces on the 1841. The interface connected to the provider shows OK as far as errors but the LAN side of the router shows steadily increasing input errors. It doesn't show any other errors, no CRC, frame, runts, giants or overruns, just generic input errors. What type of errors are those? Nothing is being logged on the console.

I moved the connection to another switch ports and the errors continue. I switched it down to 10MB and also changed the switch and the errors slow down but don't stop. Interestingly, the switch side never shows any errors. What can I do here? I guess it can be a bad interface but that is such a rare thing that I am hesitant to replace the router.

Thanks in advance.

Joseph W. Doherty · ‎11-05-2011

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

Could you post the interface stats for this interface? (I.e. sh int x)

tato386 · ‎11-06-2011

Here is what it looks like right now. I hardcoded 10/half only because this seems to get me the "least bad" performance. When using half duplex I do get CRC errors which I don't recall seeing before.

Thanks,

1841#sho int fa0/0

FastEthernet0/0 is up, line protocol is up

Hardware is Gt96k FE, address is 0017.5af4.199c (bia 0017.5af4.199c)

Description: LAN

Internet address is 192.168.1.1/24

MTU 1500 bytes, BW 10000 Kbit, DLY 1000 usec,

reliability 255/255, txload 1/255, rxload 3/255

Encapsulation ARPA, loopback not set

Keepalive set (10 sec)

Full-duplex, 10Mb/s, 100BaseTX/FX

ARP type: ARPA, ARP Timeout 04:00:00

Last input 00:00:00, output 00:00:00, output hang never

Last clearing of "show interface" counters 18:30:12

Input queue: 0/75/538565/0 (size/max/drops/flushes); Total output drops: 0

Queueing strategy: fifo

Output queue: 0/40 (size/max)

5 minute input rate 123000 bits/sec, 14 packets/sec

5 minute output rate 44000 bits/sec, 12 packets/sec

8749732 packets input, 1179749668 bytes

Received 384732 broadcasts, 0 runts, 0 giants, 0 throttles

87541 input errors, 37849 CRC, 0 frame, 0 overrun, 0 ignored

0 watchdog

0 input packets with dribble condition detected

1913022 packets output, 872588735 bytes, 0 underruns

0 output errors, 0 collisions, 0 interface resets

0 babbles, 0 late collision, 0 deferred

0 lost carrier, 0 no carrier

0 output buffer failures, 0 output buffers swapped out

Joseph W. Doherty · ‎11-06-2011

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

Be careful with your hard coding of duplex, I see your stats shows CRC errors.

As to your original question of input errors, I also see lots of input queue drops. Often indicative of the router not being able to keep up with the input stream. (Which is also likely why errors decrease when you reduce interface speed to 10 Mbps.)

Besides just too much traffic too fast for the platform, there are some other things that can cause it, but without going into those you might be also able to mitigate the problem by just increasing your input queue depth.

Kishore Chennupati · ‎11-06-2011

Hi Deigo,

Just be careful when you increase the input queue depth.

http://www.cisco.com/en/US/products/hw/routers/ps133/products_tech_note09186a0080094791.shtml

Caution:

An increase in the hold queue can have detrimental effects on network routing and response times. For protocols that use SEQ/ACK packets to determine round-trip times, do not increase the output queue. Dropping packets instead informs hosts to slow down transmissions to match available bandwidth. This is generally better than duplicate copies of the same packet within the network, which can happen with large hold queues.

HTH,

Regards,

Kishore

Joseph W. Doherty · ‎11-06-2011

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

Take note that the caution Kishore provides states: ". . . do not increase the output queue." However, there still are implications to changing input queue depth. Kishore is correct when he wrote "Just be careful when you increase the input queue depth.", but what this means in practice is try a small increase, say about an increment of 25, and see if network performance is better or worse. What we're hoping to accomplish is to mitigate transient burst drops. Sustained congestion drops would be better addressed on the upstream device's egress.

tato386 · ‎11-08-2011

Joseph:

I don't see how this can be a performance issue. Specs for the 1841 platform claim 75,000pps performance. I usually only see a few hundred pps when I use the "sho int" command. Is there a different way to measure this?

Right now I am trying to borrow a LAN analyzer to check the network. The numbers don't make sense. The "sho int" command I posted earlier showed over 500,000 input queue drops during an 18hr span on a Sunday. Nothing going on Sunday at this place. Look at the latest "sho int"

1841#sho int fa0/0

FastEthernet0/0 is up, line protocol is up

Hardware is Gt96k FE, address is 0017.5af4.199c (bia 0017.5af4.199c)

Description: LAN

Internet address is 192.168.1.1/24

MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,

reliability 255/255, txload 2/255, rxload 1/255

Encapsulation ARPA, loopback not set

Keepalive set (10 sec)

Full-duplex, 100Mb/s, 100BaseTX/FX

ARP type: ARPA, ARP Timeout 04:00:00

Last input 00:00:00, output 00:00:00, output hang never

Last clearing of "show interface" counters 01:01:37

Input queue: 0/75/1624360/0 (size/max/drops/flushes); Total output drops: 0

Queueing strategy: fifo

Output queue: 0/40 (size/max)

5 minute input rate 543000 bits/sec, 173 packets/sec

5 minute output rate 817000 bits/sec, 175 packets/sec

828140 packets input, 336254494 bytes

Received 24954 broadcasts, 0 runts, 0 giants, 0 throttles

28599 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

0 watchdog

0 input packets with dribble condition detected

635340 packets output, 379245968 bytes, 0 underruns

0 output errors, 0 collisions, 0 interface resets

0 babbles, 0 late collision, 0 deferred

0 lost carrier, 0 no carrier

0 output buffer failures, 0 output buffers swapped out

Almost 2 million input queue problems in the span of 1 hr! Notice no CRC when I set the int to full. I knew that the CRCs go away when I set to full but somehow it just seemed to work better at half duplex in spite of the CRC errors.

So back to original question. How can I tell what these input errors are if none of the error categories show error counts?

Thanks,

Diego

Joseph W. Doherty · ‎11-08-2011

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

The input rate you're looking at is a 5 minute average - drops can happen in millisecond bursts.

The 75 Kpps you mention is the spec for fast switching, process switching performance isn't documented but it's likely about 10x slower.

johnlloyd_13 · ‎11-06-2011

Hi,

Try to hardcode same speed/duplex on both ends, do a clear counters and monitor again.

If errors persists, try to swap UTP patch cable.

Sent from Cisco Technical Support iPhone App

tato386 · ‎11-06-2011

I have hardcoded several different combinations. Seems like 10/half gets me the least bad performance. I will try changing the cable when I get to the site tomorrow.

Rgds,

johnlloyd_13 · ‎11-06-2011

Try auto/auto on your 1841 FE 0/0 port.

Sent from Cisco Technical Support iPhone App

tato386 · ‎11-10-2011

Problem turned out to be an infected PC on the LAN that was flooding the router with Internet traffic. Found it with Wireshark.

Thanks to all for the input and help.

Rgds,

Diego