IfOutDiscards port channel between 6509 and Big IP F5

Unanswered Question
Jan 8th, 2014
User Badges:

Hello Cisco World and happy new year!


I'm seeing some IfOutDiscards on my port channel between 6509 and Big IP F5 LTM load balancer These are not excessive but are incrementing, there are no problems reported from users or application teams within my company, however I want to know why these are appearing.


The F5 interfaces flow control is set as F5 default "Pause RX/TX"

QOS is NOT configured on the switch


WS-C6509-E


#sh mod 8

Mod Ports Card Type                              Model              Serial No.

--- ----- -------------------------------------- ------------------ -----------

  8   48  CEF720 48 port 10/100/1000mb Ethernet  WS-X6748-GE-TX    


show interfaces counters errors mod 8

Port            Align-Err    FCS-Err   Xmit-Err    Rcv-Err UnderSize OutDiscards

Gi8/44                  0          0          0          0         0        6932

Gi8/45                  0          0          0          0         0        2947


#sh run int gi8/45


switchport

switchport access vlan 999

switchport trunk encapsulation dot1q

switchport trunk native vlan 999

switchport trunk allowed vlan 131,132,137,139,140,144-149,152,155,156,158-166

switchport trunk allowed vlan add 168-170,173,174,176,178,181,182,192-197

switchport trunk allowed vlan add 200-203,300,400

switchport mode trunk

logging event trunk-status

logging event bundle-status

no snmp trap link-status

spanning-tree portfast

channel-protocol lacp

channel-group 104 mode active

end


sh run  int Po104

Building configuration...


Current configuration : 498 bytes

!

interface Port-channel104

description

switchport

switchport access vlan 999

switchport trunk encapsulation dot1q

switchport trunk native vlan 999

switchport trunk allowed vlan 131,132,137,139,140,144-149,152,155,156,158-166

switchport trunk allowed vlan add 168-170,173,174,176,178,181,182,192-197

switchport trunk allowed vlan add 200-203,300,400

switchport mode trunk

logging event trunk-status

logging event bundle-status

spanning-tree portfast trunk

end



I'm guessing these OutDisguards are due the switch port buffer being unable to processes these packets because they are burst packets. Is there anything you can actually do? I'm thinking of changing flow control settings or maybe going up to 10GB ports, currently 1GB/sec = 1MB/ms. Its seems to me real hard to pin point these errors, could it be the application?


Any ideas are welcome!

  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 0 (0 ratings)
Loading.
Leo Laohoo Wed, 01/08/2014 - 03:20
User Badges:
  • Super Gold, 25000 points or more
  • Hall of Fame,

    The Hall of Fame designation is a lifetime achievement award based on significant overall achievements in the community. 

  • Cisco Designated VIP,

    2017 LAN, Wireless

Please post the output to the following commands:

1. sh interface ;
2. sh counter interface

Sent from Cisco Technical Support Wii App

stephendrkw Wed, 01/08/2014 - 03:24
User Badges:

Hi Leo,


show int gi8/45

GigabitEthernet8/45 is up, line protocol is up (connected)

  Hardware is C6k 1000Mb 802.3, address is 0014.f212.383c (bia 0014.f212.383c)

  Description: + ukcam00-152-f5-p1.2 ****

  MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,

     reliability 255/255, txload 8/255, rxload 6/255

  Encapsulation ARPA, loopback not set

  Keepalive set (10 sec)

  Full-duplex, 1000Mb/s, media type is 10/100/1000BaseT

  input flow-control is off, output flow-control is off

  Clock mode is auto

  ARP type: ARPA, ARP Timeout 04:00:00

  Last input never, output 00:00:37, output hang never

  Last clearing of "show interface" counters 44w3d

  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 2947

  Queueing strategy: fifo

  Output queue: 0/40 (size/max)

  5 minute input rate 27384000 bits/sec, 6596 packets/sec

  5 minute output rate 35124000 bits/sec, 8142 packets/sec

     180327430375 packets input, 105547436134334 bytes, 0 no buffer

     Received 896895 broadcasts (896895 multicasts)

     0 runts, 0 giants, 0 throttles

     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

     0 watchdog, 0 multicast, 0 pause input

     0 input packets with dribble condition detected

     174813418302 packets output, 106004071814675 bytes, 0 underruns

     0 output errors, 0 collisions, 1 interface resets

     0 babbles, 0 late collision, 0 deferred

     0 lost carrier, 0 no carrier, 0 PAUSE output

     0 output buffer failures, 0 output buffers swapped out


show int gi8/46

GigabitEthernet8/46 is up, line protocol is up (connected)

  Hardware is C6k 1000Mb 802.3, address is 0014.f212.383d (bia 0014.f212.383d)

  Description: + ukcam00-152-f5-mgmt ****

  MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,

     reliability 255/255, txload 1/255, rxload 1/255

  Encapsulation ARPA, loopback not set

  Keepalive set (10 sec)

  Full-duplex, 1000Mb/s, media type is 10/100/1000BaseT

  input flow-control is off, output flow-control is on

  Clock mode is auto

  ARP type: ARPA, ARP Timeout 04:00:00

  Last input never, output 00:00:40, output hang never

  Last clearing of "show interface" counters 44w3d

  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0

  Queueing strategy: fifo

  Output queue: 0/40 (size/max)

  5 minute input rate 56000 bits/sec, 19 packets/sec

  5 minute output rate 32000 bits/sec, 29 packets/sec

     495852169 packets input, 227906519062 bytes, 0 no buffer

     Received 15655 broadcasts (0 multicasts)

     0 runts, 0 giants, 0 throttles

     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

     0 watchdog, 0 multicast, 0 pause input

     0 input packets with dribble condition detected

     699746077 packets output, 111032329919 bytes, 0 underruns

     0 output errors, 0 collisions, 0 interface resets

     0 babbles, 0 late collision, 0 deferred

     0 lost carrier, 0 no carrier, 0 PAUSE output

     0 output buffer failures, 0 output buffers swapped out


show counters interface gi8/45

64 bit counters:

0.                      rxHCTotalPkts = 204061683459

1.                      txHCTotalPkts = 206266009005

2.                    rxHCUnicastPkts = 204060746098

3.                    txHCUnicastPkts = 204276029087

4.                  rxHCMulticastPkts = 937281

5.                  txHCMulticastPkts = 1492369568

6.                  rxHCBroadcastPkts = 14

7.                  txHCBroadcastPkts = 497610359

8.                         rxHCOctets = 108601624334924

9.                         txHCOctets = 109723719767841

10.                 rxTxHCPkts64Octets = 1050607

11.            rxTxHCPkts65to127Octets = 171869952815

12.           rxTxHCPkts128to255Octets = 69589864097

13.           rxTxHCPkts256to511Octets = 24907816728

14.          rxTxHCpkts512to1023Octets = 35465418484

15.         rxTxHCpkts1024to1518Octets = 18853492233

16.                    txHCTrunkFrames = 206263939097

17.                    rxHCTrunkFrames = 204060746179

18.                     rxHCDropEvents = 0


32 bit counters:

0.                   rxCRCAlignErrors = 0

1.                   rxUndersizedPkts = 0

2.                    rxOversizedPkts = 0

3.                     rxFragmentPkts = 0

4.                          rxJabbers = 0

5.                       txCollisions = 0

6.                         ifInErrors = 0

7.                        ifOutErrors = 0

8.                       ifInDiscards = 0

9.                  ifInUnknownProtos = 0

10.                      ifOutDiscards = 3841069

11.            txDelayExceededDiscards = 0

12.                              txCRC = 0

13.                         linkChange = 23

14.                   wrongEncapFrames = 0

All Port Counters

1.                          InPackets = 204061631261

2.                           InOctets = 108601600839051

3.                        InUcastPkts = 204060693898

4.                        InMcastPkts = 937281

5.                        InBcastPkts = 14

6.                         OutPackets = 206265938925

7.                          OutOctets = 109723689299635

8.                       OutUcastPkts = 204275959344

9.                       OutMcastPkts = 1492369279

10.                       OutBcastPkts = 497610311

11.                           AlignErr = 0

12.                             FCSErr = 0

13.                            XmitErr = 0

14.                             RcvErr = 0

15.                          UnderSize = 0

16.                          SingleCol = 0

17.                           MultiCol = 0

18.                            LateCol = 0

19.                       ExcessiveCol = 0

20.                       CarrierSense = 0

21.                              Runts = 0

22.                             Giants = 0

23.                         InDiscards = 0

24.                        OutDiscards = 3841069

25.                           InErrors = 0

26.                          OutErrors = 0

27.                    InUnknownProtos = 0

28.                              txCRC = 0

29.                      TrunkFramesTx = 206263946312

30.                      TrunkFramesRx = 204060751665

31.                         WrongEncap = 0

32.     Broadcast_suppression_discards = 0

33.     Multicast_suppression_discards = 0

34.       Unicast_suppression_discards = 0

35.                 rxTxHCPkts64Octets = 1050607

36.            rxTxHCPkts65to127Octets = 171869957437

37.           rxTxHCPkts128to255Octets = 69589867525

38.           rxTxHCPkts256to511Octets = 24907817677

39.          rxTxHCpkts512to1023Octets = 35465419496

40.         rxTxHCpkts1024to1518Octets = 18853493869

41.                         DropEvents = 0

42.                     CRCAlignErrors = 0

43.                     UndersizedPkts = 0

44.                      OversizedPkts = 0

45.                       FragmentPkts = 0

46.                            Jabbers = 0

47.                         Collisions = 0

48.              DelayExceededDiscards = 0

49.                        bpduOutlost = 0

50.                        qos0Outlost = 3841069

51.                        qos1Outlost = 0

52.                        qos2Outlost = 0

53.                        qos3Outlost = 0

54.                        qos4Outlost = 0

55.                        qos5Outlost = 0

56.                        qos6Outlost = 0

57.                        qos7Outlost = 0

58.                        qos8Outlost = 0

59.                        qos9Outlost = 0

60.                       qos10Outlost = 0

61.                       qos11Outlost = 0

62.                       qos12Outlost = 0

63.                       qos13Outlost = 0

64.                       qos14Outlost = 0

65.                       qos15Outlost = 0

66.                       qos16Outlost = 0

67.                       qos17Outlost = 0

68.                       qos18Outlost = 0

69.                       qos19Outlost = 0

70.                       qos20Outlost = 0

71.                       qos21Outlost = 0

72.                       qos22Outlost = 0

73.                       qos23Outlost = 0

74.                       qos24Outlost = 0

75.                       qos25Outlost = 0

76.                       qos26Outlost = 0

77.                       qos27Outlost = 0

78.                       qos28Outlost = 0

79.                       qos29Outlost = 0

80.                       qos30Outlost = 0

81.                       qos31Outlost = 0

82.                    bpduCbicOutlost = 0

83.                    qos0CbicOutlost = 0

84.                    qos1CbicOutlost = 0

85.                    qos2CbicOutlost = 0

86.                    qos3CbicOutlost = 0

87.                         bpduInlost = 0

88.                         qos0Inlost = 0

89.                         qos1Inlost = 0

90.                         qos2Inlost = 0

91.                         qos3Inlost = 0

92.                         qos4Inlost = 0

93.                         qos5Inlost = 0

94.                         qos6Inlost = 0

95.                         qos7Inlost = 0

96.                         qos8Inlost = 0

97.                         qos9Inlost = 0

98.                        qos10Inlost = 0

99.                        qos11Inlost = 0

100.                        qos12Inlost = 0

101.                        qos13Inlost = 0

102.                        qos14Inlost = 0

103.                        qos15Inlost = 0

104.                        qos16Inlost = 0

105.                        qos17Inlost = 0

106.                        qos18Inlost = 0

107.                        qos19Inlost = 0

108.                        qos20Inlost = 0

109.                        qos21Inlost = 0

110.                        qos22Inlost = 0

111.                        qos23Inlost = 0

112.                        qos24Inlost = 0

113.                        qos25Inlost = 0

114.                        qos26Inlost = 0

115.                        qos27Inlost = 0

116.                        qos28Inlost = 0

117.                        qos29Inlost = 0

118.                        qos30Inlost = 0

119.                        qos31Inlost = 0

120.                         pqueInlost = 0

121.                           Overruns = 0

122.                           maxIndex = 0


#show counters interface gi8/46

64 bit counters:

0.                      rxHCTotalPkts = 533905524

1.                      txHCTotalPkts = 761774290

2.                    rxHCUnicastPkts = 533885917

3.                    txHCUnicastPkts = 467174379

4.                  rxHCMulticastPkts = 144

5.                  txHCMulticastPkts = 285632202

6.                  rxHCBroadcastPkts = 19455

7.                  txHCBroadcastPkts = 8967719

8.                         rxHCOctets = 237665040716

9.                         txHCOctets = 128122576643

10.                 rxTxHCPkts64Octets = 41653443

11.            rxTxHCPkts65to127Octets = 442419759

12.           rxTxHCPkts128to255Octets = 661273707

13.           rxTxHCPkts256to511Octets = 26779075

14.          rxTxHCpkts512to1023Octets = 20848192

15.         rxTxHCpkts1024to1518Octets = 102705646

16.                    txHCTrunkFrames = 8

17.                    rxHCTrunkFrames = 8

18.                     rxHCDropEvents = 0


32 bit counters:

0.                   rxCRCAlignErrors = 0

1.                   rxUndersizedPkts = 0

2.                    rxOversizedPkts = 0

3.                     rxFragmentPkts = 0

4.                          rxJabbers = 0

5.                       txCollisions = 0

6.                         ifInErrors = 0

7.                        ifOutErrors = 0

8.                       ifInDiscards = 0

9.                  ifInUnknownProtos = 0

10.                      ifOutDiscards = 0

11.            txDelayExceededDiscards = 0

12.                              txCRC = 0

13.                         linkChange = 13

14.                   wrongEncapFrames = 0

All Port Counters

1.                          InPackets = 533905384

2.                           InOctets = 237664982447

3.                        InUcastPkts = 533885777

4.                        InMcastPkts = 144

5.                        InBcastPkts = 19455

6.                         OutPackets = 761774098

7.                          OutOctets = 128122551388

8.                       OutUcastPkts = 467174254

9.                       OutMcastPkts = 285632135

10.                       OutBcastPkts = 8967719

11.                           AlignErr = 0

12.                             FCSErr = 0

13.                            XmitErr = 0

14.                             RcvErr = 0

15.                          UnderSize = 0

16.                          SingleCol = 0

17.                           MultiCol = 0

18.                            LateCol = 0

19.                       ExcessiveCol = 0

20.                       CarrierSense = 0

21.                              Runts = 0

22.                             Giants = 0

23.                         InDiscards = 0

24.                        OutDiscards = 0

25.                           InErrors = 0

26.                          OutErrors = 0

27.                    InUnknownProtos = 0

28.                              txCRC = 0

29.                      TrunkFramesTx = 8

30.                      TrunkFramesRx = 8

31.                         WrongEncap = 0

32.     Broadcast_suppression_discards = 0

33.     Multicast_suppression_discards = 0

34.       Unicast_suppression_discards = 0

35.                 rxTxHCPkts64Octets = 41653437

36.            rxTxHCPkts65to127Octets = 442419605

37.           rxTxHCPkts128to255Octets = 661273562

38.           rxTxHCPkts256to511Octets = 26779075

39.          rxTxHCpkts512to1023Octets = 20848188

40.         rxTxHCpkts1024to1518Octets = 102705623

41.                         DropEvents = 0

42.                     CRCAlignErrors = 0

43.                     UndersizedPkts = 0

44.                      OversizedPkts = 0

45.                       FragmentPkts = 0

46.                            Jabbers = 0

47.                         Collisions = 0

48.              DelayExceededDiscards = 0

49.                        bpduOutlost = 0

50.                        qos0Outlost = 0

51.                        qos1Outlost = 0

52.                        qos2Outlost = 0

53.                        qos3Outlost = 0

54.                        qos4Outlost = 0

55.                        qos5Outlost = 0

56.                        qos6Outlost = 0

57.                        qos7Outlost = 0

58.                        qos8Outlost = 0

59.                        qos9Outlost = 0

60.                       qos10Outlost = 0

61.                       qos11Outlost = 0

62.                       qos12Outlost = 0

63.                       qos13Outlost = 0

64.                       qos14Outlost = 0

65.                       qos15Outlost = 0

66.                       qos16Outlost = 0

67.                       qos17Outlost = 0

68.                       qos18Outlost = 0

69.                       qos19Outlost = 0

70.                       qos20Outlost = 0

71.                       qos21Outlost = 0

72.                       qos22Outlost = 0

73.                       qos23Outlost = 0

74.                       qos24Outlost = 0

75.                       qos25Outlost = 0

76.                       qos26Outlost = 0

77.                       qos27Outlost = 0

78.                       qos28Outlost = 0

79.                       qos29Outlost = 0

80.                       qos30Outlost = 0

81.                       qos31Outlost = 0

82.                    bpduCbicOutlost = 0

83.                    qos0CbicOutlost = 0

84.                    qos1CbicOutlost = 0

85.                    qos2CbicOutlost = 0

86.                    qos3CbicOutlost = 0

87.                         bpduInlost = 0

88.                         qos0Inlost = 0

89.                         qos1Inlost = 0

90.                         qos2Inlost = 0

91.                         qos3Inlost = 0

92.                         qos4Inlost = 0

93.                         qos5Inlost = 0

94.                         qos6Inlost = 0

95.                         qos7Inlost = 0

96.                         qos8Inlost = 0

97.                         qos9Inlost = 0

98.                        qos10Inlost = 0

99.                        qos11Inlost = 0

100.                        qos12Inlost = 0

101.                        qos13Inlost = 0

102.                        qos14Inlost = 0

103.                        qos15Inlost = 0

104.                        qos16Inlost = 0

105.                        qos17Inlost = 0

106.                        qos18Inlost = 0

107.                        qos19Inlost = 0

108.                        qos20Inlost = 0

109.                        qos21Inlost = 0

110.                        qos22Inlost = 0

111.                        qos23Inlost = 0

112.                        qos24Inlost = 0

113.                        qos25Inlost = 0

114.                        qos26Inlost = 0

115.                        qos27Inlost = 0

116.                        qos28Inlost = 0

117.                        qos29Inlost = 0

118.                        qos30Inlost = 0

119.                        qos31Inlost = 0

120.                         pqueInlost = 0

121.                           Overruns = 0

122.                           maxIndex = 0

Leo Laohoo Wed, 01/08/2014 - 14:18
User Badges:
  • Super Gold, 25000 points or more
  • Hall of Fame,

    The Hall of Fame designation is a lifetime achievement award based on significant overall achievements in the community. 

  • Cisco Designated VIP,

    2017 LAN, Wireless

qos0Outlost = 3841069


Thanks for the outputs.


I'm no expert in QoS but Gi8/45 is dropping packets because QoS queue 0 is overflowing.  There is no noticable traffic or errors with Gi8/46.

stephendrkw Thu, 01/09/2014 - 01:36
User Badges:

With these "Oulost" counters, I am thinking rather the ports sharing the same 2 x 20Gb ASIC fabric on the 48 port line is to move the ports on different cards.


Its actually 4 port channel x 1GB


8/45

8/46

9/19

9/41


9/19 and 9/41 have the same problem


We only have 3 cards though.


What do you think.

Leo Laohoo Thu, 01/09/2014 - 01:42
User Badges:
  • Super Gold, 25000 points or more
  • Hall of Fame,

    The Hall of Fame designation is a lifetime achievement award based on significant overall achievements in the community. 

  • Cisco Designated VIP,

    2017 LAN, Wireless

You are using 6748 line card.  This line card is designed for medium speed servers.  It's got good memory buffers and should be able to handle the incoming traffic. 


I am suspecting this could be an IOS bug.


Like I said, I am no expert in the "black magic" of QoS.

stephendrkw Thu, 01/09/2014 - 01:52
User Badges:

Thanks, I'll log a TAC case and update this discussion when I get some feedback from Cisco

Leo Laohoo Thu, 01/09/2014 - 01:59
User Badges:
  • Super Gold, 25000 points or more
  • Hall of Fame,

    The Hall of Fame designation is a lifetime achievement award based on significant overall achievements in the community. 

  • Cisco Designated VIP,

    2017 LAN, Wireless

Just want to make sure that you have enabled etherchannel on the F5 side? 


Can you post the 6500 command output of "sh etherchannel 104 summary"?

stephendrkw Thu, 01/09/2014 - 02:09
User Badges:

see attached F5 config


sh etherchannel 104 summary

Flags:  D - down        P - bundled in port-channel

        I - stand-alone s - suspended

        H - Hot-standby (LACP only)

        R - Layer3      S - Layer2

        U - in use      N - not in use, no aggregation

        f - failed to allocate aggregator


        M - not in use, no aggregation due to minimum links not met

        m - not in use, port not aggregated due to minimum links not met

        u - unsuitable for bundling

        d - default port


        w - waiting to be aggregated

Number of channel-groups in use: 8

Number of aggregators:           8


Group  Port-channel  Protocol    Ports

------+-------------+-----------+-----------------------------------------------

104    Po104(SU)       LACP      Gi8/44(P)      Gi8/45(P)      Gi9/19(P)

                                 Gi9/41(P)


Last applied Hash Distribution Algorithm: Fixed

Leo Laohoo Thu, 01/09/2014 - 02:11
User Badges:
  • Super Gold, 25000 points or more
  • Hall of Fame,

    The Hall of Fame designation is a lifetime achievement award based on significant overall achievements in the community. 

  • Cisco Designated VIP,

    2017 LAN, Wireless

Thanks for the output and screenshot.  Looks fine to me.

stephendrkw Thu, 01/09/2014 - 02:14
User Badges:

Thought I would show this as well, each port is configured the same


Actions

This Discussion