High discard rate

Unanswered Question

I'm getting alerted by ciscoworks DFM that one of our etherchannel interfaces on a 6509 switch has a high discard rate. This is the event:

EVENT DESCRIPTION = HighDiscardRate::Component=IF-172.17.6.253/78 [Po4] [172.16.14.149];Type=PROPVIRTUAL;InputPacketRate=1.7907492E7 PPS;MinimumDiscardRate=33333.312;DuplexMode=FULLDUPLEX;OutputPacketRate=2585.2126 PPS;DiscardThreshold=5;InputPacketDiscardPct=99.93414

When I look at the device and I do a shot interface po 4...it doesn't LOOK like I have a very high discard rate..I just cleared the counters, waited 10 mins and here is what I have:

Port-channel4 is up, line protocol is up (connected)

Hardware is EtherChannel, address is 0017.df54.1c00 (bia 0017.df54.1c00)

Internet address is 172.16.14.149/30

MTU 1500 bytes, BW 4000000 Kbit, DLY 10 usec,

reliability 255/255, txload 1/255, rxload 4/255

Encapsulation ARPA, loopback not set

Full-duplex, 1000Mb/s

input flow-control is off, output flow-control is unsupported

Members in this channel: Gi1/22 Gi1/23 Gi1/24 Gi2/1

ARP type: ARPA, ARP Timeout 04:00:00

Last input 00:00:01, output 00:00:00, output hang never

Last clearing of "show interface" counters 00:03:04

Input queue: 0/75/0/76 (size/max/drops/flushes); Total output drops: 0

Queueing strategy: fifo

Output queue: 0/40 (size/max)

5 minute input rate 74608000 bits/sec, 12972 packets/sec

5 minute output rate 2768000 bits/sec, 2582 packets/sec

L2 Switched: ucast: 407 pkt, 98366 bytes - mcast: 2073 pkt, 1052822 bytes

L3 in Switched: ucast: 814134 pkt, 806858072 bytes - mcast: 1700090 pkt, 93197

1069 bytes mcast

L3 out Switched: ucast: 474545 pkt, 66076421 bytes mcast: 24 pkt, 1680 bytes

2388058 packets input, 1651673918 bytes, 0 no buffer

Received 1615951 broadcasts, 0 runts, 0 giants, 0 throttles

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

0 input packets with dribble condition detected

447273 packets output, 62329911 bytes, 0 underruns

0 output errors, 0 collisions, 0 interface resets

0 babbles, 0 late collision, 0 deferred

0 lost carrier, 0 no carrier

0 output buffer failures, 0 output buffers swapped out

Can someone help figure out what's going on?

I have this problem too.
0 votes
  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 0 (0 ratings)
Loading.
padramas Thu, 04/24/2008 - 16:28

Hello Mike,

Yes, the packets are being discarded due to high input rate.

Input queue: 0/75/0/76 (size/max/drops/flushes)

5 minute input rate 74608000 bits/sec, 12972 packets/sec

5 minute output rate 2768000 bits/sec, 2582 packets/sec

can you please paste o/p of " show etherchannel detail "

Padmanabhan

I'm posting the output in a file as it's a little long. 74Mbits input rate would cause discards? It's a bundle of 4 1GB ports so it shouldn't have any problems handling that rate.

here's a show etherechannel 4 summary output:

Flags: D - down P - in port-channel

I - stand-alone s - suspended

H - Hot-standby (LACP only)

R - Layer3 S - Layer2

U - in use f - failed to allocate aggregator

Number of channel-groups in use: 9

Number of aggregators: 9

Group Port-channel Protocol Ports

------+-------------+-----------+-----------------------------------------------

4 Po4(RU) LACP Gi1/21(D) Gi1/22(P) Gi1/23(P) Gi1/24(P)

Gi2/1(P)

Attachment: 
shastymacnasty Fri, 04/25/2008 - 04:08

What module have you used to create the etherchannel on? We have had issues in the past with over subscribed modules and etherchannels dropping packets. Post a show mod.

Cheers

Here's a show mod:

Mod Ports Card Type Model Serial No.

--- ----- -------------------------------------- ------------------ -----------

1 24 CEF720 24 port 1000mb SFP WS-X6724-SFP SAD100609A6

2 48 CEF720 48 port 10/100/1000mb Ethernet WS-X6748-GE-TX SAL1010F9EY

5 2 Supervisor Engine 720 (Active) WS-SUP720-3B SAL1016KBPR

Mod MAC addresses Hw Fw Sw Status

--- ---------------------------------- ------ ------------ ------------ -------

1 0016.c738.4fae to 0016.c738.4fc5 2.3 12.2(14r)S5 12.2(17d)SXB Ok

2 0016.c842.6950 to 0016.c842.697f 2.3 12.2(14r)S5 12.2(17d)SXB Ok

5 0013.c43a.d7bc to 0013.c43a.d7bf 5.2 8.4(2) 12.2(17d)SXB Ok

Mod Sub-Module Model Serial Hw Status

--- --------------------------- ------------------ ------------ ------- -------

1 Centralized Forwarding Card WS-F6700-CFC SAL1016KA3J 2.0 Ok

2 Centralized Forwarding Card WS-F6700-CFC SAL1016KHTX 2.0 Ok

5 Policy Feature Card 3 WS-F6K-PFC3B SAL1016K7LR 2.3 Ok

5 MSFC3 Daughterboard WS-SUP720 SAL1016K9FP 2.5 Ok

Mod Online Diag Status

--- -------------------

1 Pass

2 Pass

5 Pass

it's connected to fiber ports on another 6509. Here's a show mod of that one (the fiber module is mod 3 on this switch):

Mod Ports Card Type Model Serial No.

--- ----- -------------------------------------- ------------------ -----------

1 48 CEF720 48 port 10/100/1000mb Ethernet WS-X6748-GE-TX SAL1032XP9U

2 48 CEF720 48 port 10/100/1000mb Ethernet WS-X6748-GE-TX SAL1033YCLB

3 24 CEF720 24 port 1000mb SFP WS-X6724-SFP SAL1031X52E

4 6 Firewall Module WS-SVC-FWM-1 SAD103309NZ

5 2 Supervisor Engine 720 (Active) WS-SUP720-3B SAL1032XD7L

6 2 Supervisor Engine 720 (Hot) WS-SUP720-3B SAL1033Y3P4

7 4 CEF720 4 port 10-Gigabit Ethernet WS-X6704-10GE SAD103205HV

8 48 CEF720 48 port 10/100/1000mb Ethernet WS-X6748-GE-TX SAL1014JEPH

Mod MAC addresses Hw Fw Sw Status

--- ---------------------------------- ------ ------------ ------------ -------

1 0018.b988.9670 to 0018.b988.969f 2.4 12.2(14r)S5 12.2(18)SXF5 Ok

2 0018.b988.dd80 to 0018.b988.ddaf 2.4 12.2(14r)S5 12.2(18)SXF5 Ok

3 000a.b8cf.259c to 000a.b8cf.25b3 2.4 12.2(14r)S5 12.2(18)SXF5 Ok

4 0019.0628.1ad0 to 0019.0628.1ad7 4.0 7.2(1) 3.1(3) Ok

5 0017.5a34.9a24 to 0017.5a34.9a27 5.2 8.4(2) 12.2(18)SXF5 Ok

6 0017.5a34.a164 to 0017.5a34.a167 5.2 8.4(2) 12.2(18)SXF5 Ok

7 0019.0627.b708 to 0019.0627.b70b 2.4 12.2(14r)S5 12.2(18)SXF5 Ok

8 0017.9431.83b0 to 0017.9431.83df 2.3 12.2(14r)S5 12.2(18)SXF5 Ok

Mod Sub-Module Model Serial Hw Status

---- --------------------------- ------------------ ----------- ------- -------

1 Centralized Forwarding Card WS-F6700-CFC SAL1029VF2E 2.0 Ok

2 Centralized Forwarding Card WS-F6700-CFC SAL1028UZ5W 2.0 Ok

3 Centralized Forwarding Card WS-F6700-CFC SAL1027UCUS 2.0 Ok

5 Policy Feature Card 3 WS-F6K-PFC3B SAL1031WZP1 2.3 Ok

5 MSFC3 Daughterboard WS-SUP720 SAL1031WVKZ 2.5 Ok

6 Policy Feature Card 3 WS-F6K-PFC3B SAL1032XRCD 2.3 Ok

6 MSFC3 Daughterboard WS-SUP720 SAL1032XKV5 2.5 Ok

7 Centralized Forwarding Card WS-F6700-CFC SAL1029VWRW 2.0 Ok

8 Centralized Forwarding Card WS-F6700-CFC SAD101503DX 2.0 Ok

Mod Online Diag Status

---- -------------------

1 Pass

2 Pass

3 Pass

4 Pass

5 Pass

6 Pass

7 Pass

8 Pass

shastymacnasty Fri, 04/25/2008 - 06:42

From past experience I would always try and create etherchannels over modules or at least have each member spread across the module, the 6748 mod has a 1.2:1 oversubscription ratio (I might be corrected) where as the 6724 should have no oversubscription, whether this is your problem Im not sure? Hopefully one of the NetPros can provide some input...

goodluck

The etherchannel is on the 6724 module on both sides. Even if it was over subscription...70MB and even 110MB of traffic seems like a very small amount to get packet drops on.

The other alert I've started getting recently on the same etherchannel (and on ports associated with it) is high error rate...but looking at the interface, it doesn't really look like there are any errors on it...;/

padramas Fri, 04/25/2008 - 20:30

Hello Mike,

Last clearing of "show interface" counters 00:03:04

Received 1615951 broadcasts, 0 runts, 0 giants, 0 throttles

Even though the raw traffic rate is small compared to line rate, you receive high broadcast packets that are punted to CPU in small time frame.

Check for any mis-configurations and the following link provides more information on DFM error message and threshold configuration

Faults and Exceptions Diagnosed by DFM

under section ""Performance Exception ""

http://www.cisco.com/en/US/docs/net_mgmt/ciscoworks_device_fault_manager/1.1/user/guide/faults.html#wp1017965

HTH

Padmanabhan

Actions

This Discussion