04-24-2008 11:42 AM - edited 03-05-2019 10:35 PM
I'm getting alerted by ciscoworks DFM that one of our etherchannel interfaces on a 6509 switch has a high discard rate. This is the event:
EVENT DESCRIPTION = HighDiscardRate::Component=IF-172.17.6.253/78 [Po4] [172.16.14.149];Type=PROPVIRTUAL;InputPacketRate=1.7907492E7 PPS;MinimumDiscardRate=33333.312;DuplexMode=FULLDUPLEX;OutputPacketRate=2585.2126 PPS;DiscardThreshold=5;InputPacketDiscardPct=99.93414
When I look at the device and I do a shot interface po 4...it doesn't LOOK like I have a very high discard rate..I just cleared the counters, waited 10 mins and here is what I have:
Port-channel4 is up, line protocol is up (connected)
Hardware is EtherChannel, address is 0017.df54.1c00 (bia 0017.df54.1c00)
Internet address is 172.16.14.149/30
MTU 1500 bytes, BW 4000000 Kbit, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 4/255
Encapsulation ARPA, loopback not set
Full-duplex, 1000Mb/s
input flow-control is off, output flow-control is unsupported
Members in this channel: Gi1/22 Gi1/23 Gi1/24 Gi2/1
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:01, output 00:00:00, output hang never
Last clearing of "show interface" counters 00:03:04
Input queue: 0/75/0/76 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 74608000 bits/sec, 12972 packets/sec
5 minute output rate 2768000 bits/sec, 2582 packets/sec
L2 Switched: ucast: 407 pkt, 98366 bytes - mcast: 2073 pkt, 1052822 bytes
L3 in Switched: ucast: 814134 pkt, 806858072 bytes - mcast: 1700090 pkt, 93197
1069 bytes mcast
L3 out Switched: ucast: 474545 pkt, 66076421 bytes mcast: 24 pkt, 1680 bytes
2388058 packets input, 1651673918 bytes, 0 no buffer
Received 1615951 broadcasts, 0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 input packets with dribble condition detected
447273 packets output, 62329911 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out
Can someone help figure out what's going on?
04-24-2008 04:28 PM
Hello Mike,
Yes, the packets are being discarded due to high input rate.
Input queue: 0/75/0/76 (size/max/drops/flushes)
5 minute input rate 74608000 bits/sec, 12972 packets/sec
5 minute output rate 2768000 bits/sec, 2582 packets/sec
can you please paste o/p of " show etherchannel detail "
Padmanabhan
04-24-2008 05:43 PM
I'm posting the output in a file as it's a little long. 74Mbits input rate would cause discards? It's a bundle of 4 1GB ports so it shouldn't have any problems handling that rate.
here's a show etherechannel 4 summary output:
Flags: D - down P - in port-channel
I - stand-alone s - suspended
H - Hot-standby (LACP only)
R - Layer3 S - Layer2
U - in use f - failed to allocate aggregator
Number of channel-groups in use: 9
Number of aggregators: 9
Group Port-channel Protocol Ports
------+-------------+-----------+-----------------------------------------------
4 Po4(RU) LACP Gi1/21(D) Gi1/22(P) Gi1/23(P) Gi1/24(P)
Gi2/1(P)
04-25-2008 04:08 AM
What module have you used to create the etherchannel on? We have had issues in the past with over subscribed modules and etherchannels dropping packets. Post a show mod.
Cheers
04-25-2008 05:25 AM
Here's a show mod:
Mod Ports Card Type Model Serial No.
--- ----- -------------------------------------- ------------------ -----------
1 24 CEF720 24 port 1000mb SFP WS-X6724-SFP SAD100609A6
2 48 CEF720 48 port 10/100/1000mb Ethernet WS-X6748-GE-TX SAL1010F9EY
5 2 Supervisor Engine 720 (Active) WS-SUP720-3B SAL1016KBPR
Mod MAC addresses Hw Fw Sw Status
--- ---------------------------------- ------ ------------ ------------ -------
1 0016.c738.4fae to 0016.c738.4fc5 2.3 12.2(14r)S5 12.2(17d)SXB Ok
2 0016.c842.6950 to 0016.c842.697f 2.3 12.2(14r)S5 12.2(17d)SXB Ok
5 0013.c43a.d7bc to 0013.c43a.d7bf 5.2 8.4(2) 12.2(17d)SXB Ok
Mod Sub-Module Model Serial Hw Status
--- --------------------------- ------------------ ------------ ------- -------
1 Centralized Forwarding Card WS-F6700-CFC SAL1016KA3J 2.0 Ok
2 Centralized Forwarding Card WS-F6700-CFC SAL1016KHTX 2.0 Ok
5 Policy Feature Card 3 WS-F6K-PFC3B SAL1016K7LR 2.3 Ok
5 MSFC3 Daughterboard WS-SUP720 SAL1016K9FP 2.5 Ok
Mod Online Diag Status
--- -------------------
1 Pass
2 Pass
5 Pass
04-25-2008 05:59 AM
Forgot to mention...the etherchannel is made up of 3 ports on the fiber module.
Also I've been getting the same alerts on other etherchannels (etherchannel 6), which is made up of 2 fiber ports, and it's averaging about 110MB of traffic through it.
04-25-2008 06:20 AM
What are the etherchannels connected too?
04-25-2008 06:29 AM
it's connected to fiber ports on another 6509. Here's a show mod of that one (the fiber module is mod 3 on this switch):
Mod Ports Card Type Model Serial No.
--- ----- -------------------------------------- ------------------ -----------
1 48 CEF720 48 port 10/100/1000mb Ethernet WS-X6748-GE-TX SAL1032XP9U
2 48 CEF720 48 port 10/100/1000mb Ethernet WS-X6748-GE-TX SAL1033YCLB
3 24 CEF720 24 port 1000mb SFP WS-X6724-SFP SAL1031X52E
4 6 Firewall Module WS-SVC-FWM-1 SAD103309NZ
5 2 Supervisor Engine 720 (Active) WS-SUP720-3B SAL1032XD7L
6 2 Supervisor Engine 720 (Hot) WS-SUP720-3B SAL1033Y3P4
7 4 CEF720 4 port 10-Gigabit Ethernet WS-X6704-10GE SAD103205HV
8 48 CEF720 48 port 10/100/1000mb Ethernet WS-X6748-GE-TX SAL1014JEPH
Mod MAC addresses Hw Fw Sw Status
--- ---------------------------------- ------ ------------ ------------ -------
1 0018.b988.9670 to 0018.b988.969f 2.4 12.2(14r)S5 12.2(18)SXF5 Ok
2 0018.b988.dd80 to 0018.b988.ddaf 2.4 12.2(14r)S5 12.2(18)SXF5 Ok
3 000a.b8cf.259c to 000a.b8cf.25b3 2.4 12.2(14r)S5 12.2(18)SXF5 Ok
4 0019.0628.1ad0 to 0019.0628.1ad7 4.0 7.2(1) 3.1(3) Ok
5 0017.5a34.9a24 to 0017.5a34.9a27 5.2 8.4(2) 12.2(18)SXF5 Ok
6 0017.5a34.a164 to 0017.5a34.a167 5.2 8.4(2) 12.2(18)SXF5 Ok
7 0019.0627.b708 to 0019.0627.b70b 2.4 12.2(14r)S5 12.2(18)SXF5 Ok
8 0017.9431.83b0 to 0017.9431.83df 2.3 12.2(14r)S5 12.2(18)SXF5 Ok
Mod Sub-Module Model Serial Hw Status
---- --------------------------- ------------------ ----------- ------- -------
1 Centralized Forwarding Card WS-F6700-CFC SAL1029VF2E 2.0 Ok
2 Centralized Forwarding Card WS-F6700-CFC SAL1028UZ5W 2.0 Ok
3 Centralized Forwarding Card WS-F6700-CFC SAL1027UCUS 2.0 Ok
5 Policy Feature Card 3 WS-F6K-PFC3B SAL1031WZP1 2.3 Ok
5 MSFC3 Daughterboard WS-SUP720 SAL1031WVKZ 2.5 Ok
6 Policy Feature Card 3 WS-F6K-PFC3B SAL1032XRCD 2.3 Ok
6 MSFC3 Daughterboard WS-SUP720 SAL1032XKV5 2.5 Ok
7 Centralized Forwarding Card WS-F6700-CFC SAL1029VWRW 2.0 Ok
8 Centralized Forwarding Card WS-F6700-CFC SAD101503DX 2.0 Ok
Mod Online Diag Status
---- -------------------
1 Pass
2 Pass
3 Pass
4 Pass
5 Pass
6 Pass
7 Pass
8 Pass
04-25-2008 06:42 AM
From past experience I would always try and create etherchannels over modules or at least have each member spread across the module, the 6748 mod has a 1.2:1 oversubscription ratio (I might be corrected) where as the 6724 should have no oversubscription, whether this is your problem Im not sure? Hopefully one of the NetPros can provide some input...
goodluck
04-25-2008 06:53 AM
The etherchannel is on the 6724 module on both sides. Even if it was over subscription...70MB and even 110MB of traffic seems like a very small amount to get packet drops on.
The other alert I've started getting recently on the same etherchannel (and on ports associated with it) is high error rate...but looking at the interface, it doesn't really look like there are any errors on it...;/
04-25-2008 08:30 PM
Hello Mike,
Last clearing of "show interface" counters 00:03:04
Received 1615951 broadcasts, 0 runts, 0 giants, 0 throttles
Even though the raw traffic rate is small compared to line rate, you receive high broadcast packets that are punted to CPU in small time frame.
Check for any mis-configurations and the following link provides more information on DFM error message and threshold configuration
Faults and Exceptions Diagnosed by DFM
under section ""Performance Exception ""
HTH
Padmanabhan
04-26-2008 07:13 AM
Doesn't that broadcast counter include multicast? If it does that's normal because we use multicast very heavily...if not then I'd have to see what all this broadcast traffic actually is.
12-29-2010 08:01 AM
Message deleted
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide