Who Me Too'd this topic

Greetings!

We have two Nexus 5020s, running ver. 5.0(3)N1(1b), with Nexus 2148T Fabric Extenders attached.  I have noticed that on all active N5020 interfaces, as well as some of the 2148T interfaces, the jumbo packets counters are incrementing.  However, using Wireshark to capture and analyze packets on one of the interfaces, the largest packets are 1518 bytes (port on N2148T, operating as an 802.1Q trunk, with multiple VLANs configured).

Here's an example of the 'show interface' command:

TesoroDCRwBSwA# sh int eth 113/1/5
Ethernet113/1/5 is up
  Hardware: 1000 Ethernet, address: 000d.ecd5.c4c6 (bia 000d.ecd5.c4c6)
  Description: 99vmwaresrv14 VMNet
  MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
     reliability 255/255, txload 3/255, rxload 1/255
  Encapsulation ARPA
  Port mode is trunk
  full-duplex, 1000 Mb/s
  Beacon is turned off
  Input flow-control is off, output flow-control is on
  Switchport monitor is off
  EtherType is 0x8100
  Last link flapped 5week(s) 2day(s)
  Last clearing of "show interface" counters 1d00h
  30 seconds input rate 1593312 bits/sec, 199164 bytes/sec, 439 packets/sec
  30 seconds output rate 6518776 bits/sec, 814847 bytes/sec, 772 packets/sec
  Load-Interval #2: 5 minute (300 seconds)
    input rate 1.77 Mbps, 615 pps; output rate 14.47 Mbps, 1.40 Kpps
  RX
    20679951 unicast packets  23649 multicast packets  19307 broadcast packets
    20722905 input packets  8592861396 bytes
    3153398 jumbo packets  0 storm suppression packets   <========================  Jumbo packets Received
    0 runts  0 giants  0 CRC  0 no buffer
    0 input error  0 short frame  0 overrun   0 underrun  0 ignored
    0 watchdog  0 bad etype drop  0 bad proto drop  0 if down drop
    0 input with dribble  0 input discard
    0 Rx pause
  TX
    34549861 unicast packets  1822766 multicast packets  2247607 broadcast packets
    38620234 output packets  38534039898 bytes
    17837212 jumbo packets    <========================  Jumbo packets Transmitted
    0 output errors  0 collision  0 deferred  0 late collision
    0 lost carrier  0 no carrier  0 babble
    0 Tx pause
  0 interface resets

TesoroDCRwBSwA#

However, issuing the 'show counters' command for the same port shows zero packets above 1518 bytes, in either direction:

TesoroDCRwBSwA# sh int eth113/1/5 counters detailed all
Ethernet113/1/5
  64 bit counters:
   0.                      rxHCTotalPkts = 20727315
   1.                       txHCTotalPks = 38628399
   2.                    rxHCUnicastPkts = 20684358
   3.                    txHCUnicastPkts = 34557774
   4.                  rxHCMulticastPkts = 23649
   5.                  txHCMulticastPkts = 1822882
   6.                  rxHCBroadcastPkts = 19308
   7.                  txHCBroadcastPkts = 2247743
   8.                         rxHCOctets = 8594691743
   9.                         txHCOctets = 38543398898
  10.                 rxTxHCPkts64Octets = 4742
  11.            rxTxHCpkts65to127Octets = 21245507
  12.           rxTxHCpkts128to255Octets = 3638512
  13.           rxTxHCpkts256to511Octets = 3034053
  14.          rxTxHCpkts512to1023Octets = 2411770
  15.         rxTxHCpkts1024to1518Octets = 8025526
  16.         rxTxHCpkts1519to1548Octets = 0   <========================  Jumbo packets Received or Transmitted
  17.                    rxHCTrunkFrames = 20727315
  18.                    txHCTrunkFrames = 38625433
  19.                     rxHCDropEvents = 0

  All Port Counters:
   0.                          InPackets = 20727315
   1.                           InOctets = 8594691743
   2.                        InUcastPkts = 20684358
   3.                        InMcastPkts = 23649
   4.                        InBcastPkts = 19308
   5.                        InJumboPkts = 3154161   <========================  Jumbo packets Received
   6.                  StormSuppressPkts = 0
   7.                         OutPackets = 38628399
   8.                          OutOctets = 38543398898
   9.                       OutUcastPkts = 34557774
  10.                       OutMcastPkts = 1822882
  11.                       OutBcastPkts = 2247743
  12.                       OutJumboPkts = 17841443   <========================  Jumbo packets Transmitted
  13.                   rxHCPkts64Octets = 323
  14.              rxHCPkts65to127Octets = 11821883
  15.             rxHCPkts128to255Octets = 1769680
  16.             rxHCPkts256to511Octets = 1931991
  17.            rxHCpkts512to1023Octets = 1275158
  18.           rxHCpkts1024to1518Octets = 774119
  19.           rxHCpkts1519to1548Octets = 0   <========================  Jumbo packets Received
  20.                   txHCPkts64Octets = 4419
  21.              txHCPkts65to127Octets = 9423624
  22.             txHCPkts128to255Octets = 1868832
  23.             txHCPkts256to511Octets = 1102062
  24.            txHCpkts512to1023Octets = 1136612
  25.           txHCpkts1024to1518Octets = 7251407
  26.           txHCpkts1519to1548Octets = 0   <========================  Jumbo packets Transmitted
  27.                        ShortFrames = 0
  28.                         Collisions = 0
  29.                          SingleCol = 0
  30.                           MultiCol = 0
  31.                            LateCol = 0
  32.                       ExcessiveCol = 0
  33.                        LostCarrier = 0
  34.                          NoCarrier = 0
  35.                              Runts = 0
  36.                             Giants = 0
  37.                           InErrors = 0
  38.                          OutErrors = 0
  39.                      InputDiscards = 0
  40.                      BadEtypeDrops = 0
  41.                        IfDownDrops = 0
  42.                    InUnknownProtos = 0
  43.                           txErrors = 0
  44.                              rxCRC = 0
  45.                             Symbol = 0
  46.                          txDropped = 0
  47.                      TrunkFramesTx = 38625433
  48.                      TrunkFramesRx = 20727315
  49.                         WrongEncap = 0
  50.                            Babbles = 0
  51.                          Watchdogs = 0
  52.                                ECC = 0
  53.                           Overruns = 0
  54.                          Underruns = 0
  55.                           Dribbles = 0
  56.                           Deferred = 0
  57.                            Jabbers = 0
  58.                           NoBuffer = 0
  59.                            Ignored = 0
  60.                        bpduOutLost = 0
  61.                        cos0OutLost = 0
  62.                        cos1OutLost = 0
  63.                        cos2OutLost = 0
  64.                        cos3OutLost = 0
  65.                        cos4OutLost = 0
  66.                        cos5OutLost = 0
  67.                        cos6OutLost = 0
  68.                        cos7OutLost = 0
  69.                            RxPause = 0
  70.                            TxPause = 0
  71.                             Resets = 0
  72.                            SQETest = 0
  73.                     InLayer3Routed = 0
  74.               InLayer3RoutedOctets = 0
  75.                    OutLayer3Routed = 0
  76.              OutLayer3RoutedOctets = 0
  77.                   OutLayer3Unicast = 0
  78.             OutLayer3UnicastOctets = 0
  79.                 OutLayer3Multicast = 0
  80.           OutLayer3MulticastOctets = 0
  81.                    InLayer3Unicast = 0
  82.              InLayer3UnicastOctets = 0
  83.                  InLayer3Multicast = 0
  84.            InLayer3MulticastOctets = 0
  85.              InLayer3AverageOctets = 0
  86.             InLayer3AveragePackets = 0
  87.             OutLayer3AverageOctets = 0
  88.            OutLayer3AveragePackets = 0

TesoroDCRwBSwA#

A check of the VMWare vswitch associated with this port indicates a setting of 1500 for MTU. 

Also, all Nexus 5020 ports are showing the presence of jumbo packets.  The Nexus 5020s connect to the Fabric Extenders, as well as each other (vPC), and to the upstream Nexus 7010s.  Interestingly, the Nexus 7010s do not show ANY jumbo packets.  Here are the interface counters for one Nexus 5020 and it's upstream Nexus 7010:

TesoroDCRwBSwA# sh cdp nei int eth 1/37
Capability Codes: R - Router, T - Trans-Bridge, B - Source-Route-Bridge
                  S - Switch, H - Host, I - IGMP, r - Repeater,
                  V - VoIP-Phone, D - Remotely-Managed-Device,
                  s - Supports-STP-Dispute

Device-ID             Local Intrfce Hldtme Capability  Platform      Port ID

TesoroDCCore1(TBM13301746)Eth1/37       174    R S I s   N7K-C7010     Eth1/17
TesoroDCRwBSwA#

TesoroDCRwBSwA# sh int eth 1/37
Ethernet1/37 is up
  Hardware: 10000 Ethernet, address: 000d.ecd4.062c (bia 000d.ecd4.062c)
  Description: To TesoroDCCore1 Eth 1/17
  MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA
  Port mode is trunk
  full-duplex, 10 Gb/s, media type is 10G
  Beacon is turned off
  Input flow-control is off, output flow-control is off
  Rate mode is dedicated
  Switchport monitor is off
  EtherType is 0x8100
  Last link flapped 3week(s) 5day(s)
  Last clearing of "show interface" counters never
  30 seconds input rate 2205440 bits/sec, 275680 bytes/sec, 813 packets/sec
  30 seconds output rate 2821256 bits/sec, 352657 bytes/sec, 687 packets/sec
  Load-Interval #2: 5 minute (300 seconds)
    input rate 2.02 Mbps, 570 pps; output rate 3.55 Mbps, 715 pps
  RX
    6508572753 unicast packets  92620006 multicast packets  122377981 broadcast packets
    6723570740 input packets  5536783708240 bytes
    2925141155 jumbo packets  0 storm suppression packets   <========================  Jumbo packets Received
    0 runts  0 giants  0 CRC  0 no buffer
    0 input error  0 short frame  0 overrun   0 underrun  0 ignored
    0 watchdog  0 bad etype drop  0 bad proto drop  0 if down drop
    0 input with dribble  0 input discard
    0 Rx pause
  TX
    79841117731 unicast packets  35023627 multicast packets  24503979 broadcast packets
    79900645337 output packets  61972162794577 bytes
    27090297260 jumbo packets   <========================  Jumbo packets Transmitted
    0 output errors  0 collision  0 deferred  0 late collision
    0 lost carrier  0 no carrier  0 babble
    0 Tx pause
  4 interface resets

TesoroDCRwBSwA#

TesoroDCCore1# sh int eth 1/17
Ethernet1/17 is up
  Hardware: 10000 Ethernet, address: 0026.0b91.614c (bia 0026.0b91.614c)
  Description: To TesoroDCRowBSwA Eth1/37
  MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec
  reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA
  Port mode is trunk
  full-duplex, 10 Gb/s, media type is 10G
  Beacon is turned off
  Auto-Negotiation is turned off
  Input flow-control is off, output flow-control is off
  Rate mode is shared
  Switchport monitor is off
  EtherType is 0x8100
  Last link flapped 3week(s) 5day(s)
  Last clearing of "show interface" counters never
  30 seconds input rate 4551368 bits/sec, 926 packets/sec
  30 seconds output rate 2250952 bits/sec, 723 packets/sec
  Load-Interval #2: 5 minute (300 seconds)
    input rate 3.83 Mbps, 773 pps; output rate 1.87 Mbps, 546 pps
  RX
    2739607515 unicast packets  3559663 multicast packets  9467871 broadcast packets
    2752635049 input packets  1319670987621 bytes
    0 jumbo packets  0 storm suppression packets   <========================  Jumbo packets Received
    0 runts  0 giants  0 CRC  0 no buffer
    0 input error  0 short frame  0 overrun   0 underrun  0 ignored
    0 watchdog  0 bad etype drop  0 bad proto drop  0 if down drop
    0 input with dribble  0 input discard
    0 Rx pause
  TX
    3555239664 unicast packets  9423339 multicast packets  21514304 broadcast packets
    3586177307 output packets  3155586707204 bytes
    0 jumbo packets   <========================  Jumbo packets Transmitted
    0 output error  0 collision  0 deferred  0 late collision
    0 lost carrier  0 no carrier  0 babble  0 output discard
    0 Tx pause
  3 interface resets

TesoroDCCore1#

Clearly there is a problem, as the N7010 says it did not transmit ANY jumbo packets, yet the connected N5020 says it received almost 3 Billion!

The N7010s are running ver. 5.1(3) of Nexus OS.

Could this be a software bug in Nexus OS on the 5020?  I didn't find any ddts about this issue.

Thank you in advance for your insight.

-rb