Cisco Support Community
cancel
Showing results for 
Search instead for 
Did you mean: 
Announcements

Welcome to Cisco Support Community. We would love to have your feedback.

For an introduction to the new site, click here. If you'd prefer to explore, try our test area to get started. And see here for current known issues.

New Member

3945 input errors and overruns

Guys,

 

I have a 3945 that is seeing incrementing input errors and overruns.  Buffer seems to be mostly fine, I added more buffers just to be sure.  I read that the overrun error is usually interface hardware could not handle inbound traffic faster enough.  But the traffic rate is at around 22Mbps over 30 second average.  Not a lot for a gigabit interface.  The overrun error increment by about 1 per two thousand packets.  Also,  rx_overflow_err  is incrementing around 400 per second.

Is there a fix for this?  Thanks a lot guys!

 

#sh int g0/0
GigabitEthernet0/0 is up, line protocol is up
  Hardware is PQ3_TSEC, address is 7c69.f66c.cc80 (bia 7c69.f66c.cc80)
  Description: Stealth Internet
  Internet address is 207.251.103.122/30
  MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 5/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec) 
  Full Duplex, 1Gbps, media type is RJ45
  output flow-control is unsupported, input flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:00, output 00:00:00, output hang never
  Last clearing of "show interface" counters 00:36:20
  Input queue: 0/1000/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  30 second input rate 22862000 bits/sec, 4714 packets/sec
  30 second output rate 2893000 bits/sec, 1857 packets/sec
     10501821 packets input, 6532404997 bytes, 0 no buffer
     Received 1 broadcasts (0 IP multicasts)
     0 runts, 0 giants, 0 throttles
     4671 input errors, 0 CRC, 0 frame, 4671 overrun, 0 ignored
     0 watchdog, 0 multicast, 0 pause input
     4163926 packets output, 778284915 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out

 

#sh buffers
Buffer elements:
     1048 in free list
     599358 hits, 0 misses, 1626 created

Public buffer pools:
Small buffers, 104 bytes (total 201, permanent 200, peak 201 @ 00:11:47):
     197 in free list (50 min, 250 max allowed)
     174601 hits, 12 misses, 35 trims, 36 created
     0 failures (0 no memory)
Middle buffers, 600 bytes (total 200, permanent 200):
     198 in free list (50 min, 250 max allowed)
     924631 hits, 10 misses, 30 trims, 30 created
     0 failures (0 no memory)
Big buffers, 1536 bytes (total 200, permanent 200):
     200 in free list (50 min, 250 max allowed)
     149483 hits, 0 misses, 1 trims, 1 created
     0 failures (0 no memory)
VeryBig buffers, 4520 bytes (total 10, permanent 10, peak 11 @ 20:18:12):
     10 in free list (0 min, 100 max allowed)
     0 hits, 0 misses, 1 trims, 1 created
     0 failures (0 no memory)
Large buffers, 5024 bytes (total 10, permanent 10, peak 210 @ 00:11:25):
     10 in free list (5 min, 10 max allowed)
     0 hits, 0 misses, 18 trims, 18 created
     0 failures (0 no memory)
Huge buffers, 18024 bytes (total 5, permanent 0, peak 5 @ 20:18:12):
     5 in free list (4 min, 10 max allowed)
     0 hits, 0 misses, 14 trims, 19 created
     0 failures (0 no memory)

Interface buffer pools:
CF Small buffers, 104 bytes (total 101, permanent 100, peak 101 @ 20:18:28):
     101 in free list (100 min, 200 max allowed)
     0 hits, 0 misses, 7 trims, 8 created
     0 failures (0 no memory)
CF Middle buffers, 600 bytes (total 101, permanent 100, peak 101 @ 20:18:28):
     101 in free list (100 min, 200 max allowed)
     0 hits, 0 misses, 7 trims, 8 created
     0 failures (0 no memory)
Syslog ED Pool buffers, 600 bytes (total 133, permanent 132, peak 133 @ 20:18:07):
     101 in free list (132 min, 132 max allowed)
     168 hits, 0 misses
IPMUX SF buffers, 1500 bytes (total 500, permanent 500):
     500 in free list (0 min, 1000 max allowed)
     0 hits, 0 misses, 0 trims, 0 created
     0 failures (0 no memory)
CF Big buffers, 1536 bytes (total 26, permanent 25, peak 26 @ 20:18:28):
     26 in free list (25 min, 50 max allowed)
     0 hits, 0 misses, 7 trims, 8 created
     0 failures (0 no memory)
pak coalesce engine buffers, 1536 bytes (total 522, permanent 256, peak 522 @ 01:17:35):
     9 in free list (0 min, 512 max allowed)
     261605 hits, 136 misses, 6 trims, 272 created
     0 failures (0 no memory)
     512 max cache size, 512 in cache
     253257 hits in cache, 0 misses in cache
SEC Eng Packet buffers, 1700 bytes (total 1024, permanent 1024):
     0 in free list (0 min, 1024 max allowed)
     1024 hits, 0 fallbacks
     1024 max cache size, 1024 in cache
     0 hits in cache, 0 misses in cache
IPC buffers, 4096 bytes (total 2, permanent 2):
     1 in free list (1 min, 8 max allowed)
     1 hits, 0 fallbacks, 0 trims, 0 created
     0 failures (0 no memory)
CF VeryBig buffers, 4520 bytes (total 3, permanent 2, peak 3 @ 20:18:28):
     3 in free list (2 min, 4 max allowed)
     0 hits, 0 misses, 7 trims, 8 created
     0 failures (0 no memory)
CF Large buffers, 5024 bytes (total 2, permanent 1, peak 2 @ 20:18:28):
     2 in free list (1 min, 2 max allowed)
     0 hits, 0 misses, 7 trims, 8 created
     0 failures (0 no memory)
IPC Medium buffers, 16384 bytes (total 2, permanent 2):
     2 in free list (1 min, 8 max allowed)
     0 hits, 0 fallbacks, 0 trims, 0 created
     0 failures (0 no memory)
IPC Large buffers, 65535 bytes (total 17, permanent 16, peak 17 @ 20:18:42):
     17 in free list (16 min, 16 max allowed)
     0 hits, 0 misses, 1218 trims, 1219 created
     0 failures (0 no memory)

Header pools:
Header buffers, 0 bytes (total 768, permanent 768):
     256 in free list (128 min, 1024 max allowed)
     512 hits, 0 misses, 0 trims, 0 created
     0 failures (0 no memory)
     512 max cache size, 512 in cache
     84996 hits in cache, 0 misses in cache

Particle Clones:
     1024 clones, 0 hits, 0 misses

Public particle pools:
F/S buffers, 256 bytes (total 768, permanent 768):
     256 in free list (128 min, 1024 max allowed)
     512 hits, 0 misses, 0 trims, 0 created
     0 failures (0 no memory)
     512 max cache size, 512 in cache
     0 hits in cache, 0 misses in cache
Normal buffers, 1548 bytes (total 3840, permanent 3840):
     3840 in free list (128 min, 4096 max allowed)
     896 hits, 0 misses, 0 trims, 0 created
     0 failures (0 no memory)

Private particle pools:
HQF Particle buffers, 0 bytes (total 2000, permanent 2000):
     2000 in free list (500 min, 2000 max allowed)
     378558 hits, 0 misses, 0 trims, 0 created
     0 failures (0 no memory)
IDS SM buffers, 240 bytes (total 128, permanent 128):
     0 in free list (0 min, 128 max allowed)
     128 hits, 0 fallbacks
     128 max cache size, 128 in cache
     0 hits in cache, 0 misses in cache
SEC Eng Particle Header buffers, 256 bytes (total 1024, permanent 1024):
     0 in free list (0 min, 1024 max allowed)
     1024 hits, 0 fallbacks
     1024 max cache size, 1024 in cache
     0 hits in cache, 0 misses in cache
IPMUX particle pool buffers, 512 bytes (total 500, permanent 500):
     0 in free list (0 min, 1000 max allowed)
     500 hits, 1 misses
     1000 max cache size, 500 in cache
     0 hits in cache, 0 misses in cache
GigabitEthernet0/0 buffers, 1664 bytes (total 768, permanent 768):
     0 in free list (0 min, 768 max allowed)
     768 hits, 0 fallbacks
     768 max cache size, 511 in cache
     101226237 hits in cache, 0 misses in cache
GigabitEthernet0/1 buffers, 1664 bytes (total 768, permanent 768):
     0 in free list (0 min, 768 max allowed)
     768 hits, 0 fallbacks
     768 max cache size, 512 in cache
     248587186 hits in cache, 0 misses in cache
GigabitEthernet0/2 buffers, 1664 bytes (total 768, permanent 768):
     0 in free list (0 min, 768 max allowed)
     768 hits, 0 fallbacks
     768 max cache size, 512 in cache
     12843 hits in cache, 0 misses in cache
SEC Eng Particle buffers, 1700 bytes (total 1024, permanent 1024):
     0 in free list (0 min, 1024 max allowed)
     1024 hits, 0 misses
     1024 max cache size, 1024 in cache
     0 hits in cache, 0 misses in cache

 

#sh controller g0/0
Interface GigabitEthernet0/0
Hardware is PQUICC3 M8572E
ADDR: 0x1C5C914, FASTSEND: 0x401C114
Route Cache Flag: 0x11

= Media  RJ45 =


PHY REGISTERS:
        Display for CRC check for page:6

        PACKET_GENERATION reg_addr: 16: 0x0010
        CRC COUNTERS reg_addr: 17: 0xFF00

        page number:0

        Copper Contrl @ reg_addr: 0: 0x1140
        Copper Status @ reg_addr: 1: 0x796D
        PHY Idf1 @ reg_addr: 2: 0x0141
        PHY Idf2 @ reg_addr: 3: 0x0C97
        Autoneg Adv @ reg_addr: 4: 0x0DE1
        Autoneg Lnk Ptr abty @ reg_addr: 5: 0xC1E1
        Autoneg Exp @ reg_addr: 6: 0x000D
        Autoneg Nx Pg Tx @ reg_addr: 7: 0x2801
        Autoneg Lk Prt Nx Pg @ reg_addr: 8: 0x4E1E
        1000B-T Ctr @ reg_addr: 9: 0x0E00
        1000B-T St @ reg_addr: 10: 0x3800
        Copper Ext St @ reg_addr: 15: 0x3000
        Copper Specific Ctl1 @ reg_addr: 16: 0x6060
        Copper Specific St1 @ reg_addr: 17: 0xAC48
        Copper Interrupt En @ reg_addr: 18: 0x0000
        Copper Specific St2 @ reg_addr: 19: 0x0000
        Copper Rcv Er Ctr @ reg_addr: 21: 0x0000
        Copper Specific Ctl2 @  reg_addr: 26: 0x8040

        page number:1

        Fiber Contrl @ reg_addr: 0: 0x1140
        Fiber Status @ reg_addr: 1: 0x794D
        PHY Idf1 @ reg_addr: 2: 0x0141
        PHY Idf2 @ reg_addr: 3: 0x0C97
        Autoneg Adv @ reg_addr: 4: 0x0060
        Autoneg Lnk Ptr abty @ reg_addr: 5: 0x0000
        Autoneg Exp @ reg_addr: 6: 0x0004
        Autoneg Nx Pg Tx @ reg_addr: 7: 0x2001
        Autoneg Lk Prt Nx Pg @ reg_addr: 8: 0x0000
        Fiber Ext St @ reg_addr: 15: 0x3000
        Fiber Specific Ctl1 @ reg_addr: 16: 0x0000
        Fiber Specific St1 @ reg_addr: 17: 0x0018
        Fiber Interrupt En @ reg_addr: 18: 0x0000
        Fiber Specific St2 @ reg_addr: 19: 0x0000
        Fiber Rcv Er Ctr @ reg_addr: 21: 0x0000
        Fiber Specific Ctl2 @  reg_addr: 26: 0x8085

        page number:2

        MAC Contrl @ reg_addr: 0: 0x0040
        PHY Idf1 @ reg_addr: 2: 0x0141
        PHY Idf2 @ reg_addr: 3: 0x0C97
        MAC Specific Ctl1 @ reg_addr: 16: 0x0288
        MAC Specific St1 @ reg_addr: 17: 0x0440
        MAC Interrupt En @ reg_addr: 18: 0x0000
        MAC Specific St2 @ reg_addr: 19: 0x0000
        MAC Specific Ctl2 @  reg_addr: 26: 0x0005

MPC8500 TSEC Registers:
        IADDR0     : 0x00000000, IADDR1   : 0x00000000
        IADDR2     : 0x00000000, IADDR3   : 0x00000000
        IADDR4     : 0x00000000, IADDR5   : 0x00000000
        IADDR6     : 0x00000000, IADDR7   : 0x40000000
        GADDR0     : 0x00000000, GADDR1   : 0x00000004
        GADDR2     : 0x00001000, GADDR3   : 0x00000000
        GADDR4     : 0x01000000, GADDR5   : 0x20000000
        GADDR6     : 0x00000000, GADDR7   : 0x00000001
        IEVENT     : 0x00000000, IMASK    : 0xB3708180
        EDIS       : 0x00000000, ECNTRL   : 0x00001022
        MINFLR     : 0x00000000, PTV      : 0x00000060
        DMACTRL    : 0x000000C1, TBIPA    : 0x00000011
        TSTAT      : 0x80008000, TBDLEN   : 0x00000000
        CTBPTR     : 0x00000000, TBPTR    : 0x3CFB3818
        TBASE      : 0x3CFB32C0, OSTBD    : 0x00000000
        OSTBDP     : 0x00000000, RCTRL    : 0x00000004
        RSTAT      : 0x00000080, RBDLEN   : 0x00000000
        CRBPTR     : 0x00000000, MRBLR    : 0x00000600
        MACCFG1    : 0x0000002F, MACCFG2  : 0x00007225
        RBASE      : 0x3CFB2A80, IPGIFG   : 0x40605060
        HAFDUP     : 0x0001F037, MAXFRML  : 0x00002580
        MIIMCFG    : 0x00000015, MIIMCOM  : 0x00000000
        MIIMADD    : 0x00000300, MIIMCON  : 0x00000000
        MIIMSTAT   : 0x0000FFFF, MIIMIND  : 0x00000000
        IFSTAT     : 0x00000008, TCTRL    : 0x00000000
        ATTR       : 0x000048C0, ATTRELI  : 0x00500000
        MACSTNADDR1 : 0x80CC6CF6
        MACSTNADDR2 : 0x697C0000
        FIFO_TX_THR : 0x00000100
        FIFO_TX_STARVE : 0x000000E0
        FIFO_STARVE_SHUTOFF : 0x00000100
        FIFO_PAUSE_CTRL: 0x00000000
============RMON MIB Registers==============
        TR64=2055300, TR127=2624725
        TR255=1156119, TR511=1464021
        TR1K=3744253, TRMAX=941666
        TRMGV=0
        RBYT=2432586826, RPKT=1693178
        RFCS=0, RMCA=0
        RBCA=9, RXCF=0
        RXPF=0, RXUO=0
        RALN=0, RFLR=0
        RCDE=0, RCSE=0
        RUND=1, ROVR=0
        RFRG=0, RJBR=0
        RDRP=28887
        TBYT=1497944823, TPKT=1904299
        TMCA=1479, TBCA=6
        TXPF=0, TDFR=0
        TEDF=0, TSCL=0
        TMCL=0, TLCL=0
        TNCL=0, TDRP=0
        TJBR=0, TFCS=0
        TXCF=0, TOVR=0
        TUND=0, TFRG=0
        CAR1=0x00000000, CAR2=0x00003000
        CAM1=0xFE03FFFF, CAM2=0x000FFFFD


Software MAC Address Filter (hash:length/addr/mask/hits)
--------------------------------------------------------
  0x000:  0  ffff.ffff.ffff  0000.0000.0000         5
  0x076:  0  7c69.f66c.cc80  0000.0000.0000 101934361
  0x0C0:  0  0100.0ccc.cccc  0000.0000.0000         0
  0x0C0:  1  0180.c200.0002  0000.0000.0000         0
  0x0C5:  0  0180.c200.0007  0000.0000.0000         0
  0x0CC:  0  0180.c200.000e  0000.0000.0000         0


  Software filtered frames: 0
  Promiscuous mode: 0

Statistics:
  Rx Bytes                  2432586826   Tx Bytes                  1497945099
  Rx Good Packets              1693177   Tx Good Packets              1904301
  Rx Multicast                       0
  Rx Broadcast                       9

  Rx Bad Pkt Errors                  1   Tx Bad Pkt Errors                  0
  Rx FCS Errors                      0   Tx FCS Errors                      0
  Rx Runt Errors                     1   Tx Runt Errors                     0
  Rx Oversize Errors                 0   Tx Oversize Errors                 0
  Rx Length Errors                   0   Tx Collisions                      0
  Rx Code Errors                     0   Tx Late Collisions                 0
  Rx Dribble Errors                  0   Tx Excessive Collisions            0
                                         Tx Abort Errors                    0

Port Stopped: N

Internal Loopback Set: N

Internal Driver Information:
 throttled=0, enabled=0, disabled=0
 rx_coalesce_failed=0, rx_framing_err=0, rx_overflow_err=719619, rx_buffer_err=0
 rx_no_enp=0, rx_discard=0
 tx_one_col_err=0, tx_more_col_err=0, tx_no_enp=0, tx_deferred_err=0
 tx_underrun_err=0, tx_late_collision_err=0, tx_loss_carrier_err=0
 tx_exc_collision_err=0, tx_buff_err=0, fatal_tx_err=0

Ring Buffer Information:
 pq3_tsec_instance=0x1C5D700
 rx ring entries=256, tx ring entries=256
 rxring=0x3CFB2A80, rxr shadow=0x1C5DBA0, rx_head=120, rx_tail=0
 txring=0x3CFB32C0, txr shadow=0x13AC9C84, tx_head=217, tx_tail=217, tx_count=0
 Driver Level Counters: (Cumulative, Zeroed only at Reset)
 Rx Frames=101927547, Bytes=1258377525
 Tx Frames=48039129, Bytes=1275424370
 tx_ring_limit_min = 64 tx_ring_limit_max = 256 tx_ring_limit_current = 240

 

 

The following two commands are with in a second of each other

#sh controller g0/0 | i rx_overflow_err
 rx_coalesce_failed=0, rx_framing_err=0, rx_overflow_err=80110, rx_buffer_err=0
#sh controller g0/0 | i rx_overflow_err
 rx_coalesce_failed=0, rx_framing_err=0, rx_overflow_err=80598, rx_buffer_err=0

 

 

4 REPLIES
New Member

 Well for 22mbps you do have

 

Well for 22mbps you do have considerable packet per second around 5K. Meaning your packets are rather small, probably some UDP traffic.  I don't think you should underestimate 22mbps traffic on the 30 second average. 1 second is just so much time, when you think about the fraction of ms. 

If you have a few packets arriving at the interface on the same ms fraction, and the input 'bus' is not able to read them fast enough to put them in memory buffer, packets will have to be dropped as an input drop. You can have as much buffer memory as you want, this wont help. 

Consider this is real 'demand' traffic and depends a lot on usage pattern' You can very well get 1gbps of TCP packets without any input errors with iperf, if your packet arrive at the correct time. 

Do the same test, with UDP and random test arrivals, and very small packets, and with not so much traffic on the average you can start increasing your input errors. 

The solution by the book for this is 'port agregation' maybe you want to give that a try and see if your error count lowers or disappear. 

New Member

I agree microburst traffic

I agree microburst traffic could be the cause here.  The 3948 is rated at 982 kpps with 64 byte packets.  On one hand, we are not even at 1% of the rated capacity here based on pps.  On the other hand, 982k of 64 byte packets plus 20 bytes overhead, is about 660Mbps.  This means that if the traffic is coming in at line rate for 64 byte packets, there will be a lot of drops.  My conclusion is that the 3945 platform is not capable of processing small packets at gigabit line rate.

Super Bronze

DisclaimerThe Author of this

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

Just to add to ylingf's post, the 660 Mbps would be best case, i.e. router doing nothing but simple packet forwarding.  Cisco recommends the 3945 for up to 150 Mbps of WAN traffic.

What you might try is running the gig interface at 100, this will avoid gig bursts over running the capacity of the router.

New Member

Almost no platform is capable

Almost no platform is capable of doing this micropacket processing. Still its not common. Cisco rates their products with packets size of 64bytes which turns into very small traffic rates. In real deployment you will much larger packet sizes that 64bytes almost 8 to 10 times larger.

The end result of this, is people oversizing their routing requirements everywhere.

If you want true performance, and line rate routing either the 65XX, 76XX or 3750 are just best choice.

1065
Views
0
Helpful
4
Replies