Nexus 5000 Jumbo Packet Counters - Bug?

[ronbuchalski]

Greetings!

We have two Nexus 5020s, running ver. 5.0(3)N1(1b), with Nexus 2148T Fabric Extenders attached.  I have noticed that on all active N5020 interfaces, as well as some of the 2148T interfaces, the jumbo packets counters are incrementing.  However, using Wireshark to capture and analyze packets on one of the interfaces, the largest packets are 1518 bytes (port on N2148T, operating as an 802.1Q trunk, with multiple VLANs configured).

Here's an example of the 'show interface' command:

TesoroDCRwBSwA# sh int eth 113/1/5
Ethernet113/1/5 is up
  Hardware: 1000 Ethernet, address: 000d.ecd5.c4c6 (bia 000d.ecd5.c4c6)
  Description: 99vmwaresrv14 VMNet
  MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
     reliability 255/255, txload 3/255, rxload 1/255
  Encapsulation ARPA
  Port mode is trunk
  full-duplex, 1000 Mb/s
  Beacon is turned off
  Input flow-control is off, output flow-control is on
  Switchport monitor is off
  EtherType is 0x8100
  Last link flapped 5week(s) 2day(s)
  Last clearing of "show interface" counters 1d00h
  30 seconds input rate 1593312 bits/sec, 199164 bytes/sec, 439 packets/sec
  30 seconds output rate 6518776 bits/sec, 814847 bytes/sec, 772 packets/sec
  Load-Interval #2: 5 minute (300 seconds)
    input rate 1.77 Mbps, 615 pps; output rate 14.47 Mbps, 1.40 Kpps
  RX
    20679951 unicast packets  23649 multicast packets  19307 broadcast packets
    20722905 input packets  8592861396 bytes
    3153398 jumbo packets  0 storm suppression packets   <========================  Jumbo packets Received
    0 runts  0 giants  0 CRC  0 no buffer
    0 input error  0 short frame  0 overrun   0 underrun  0 ignored
    0 watchdog  0 bad etype drop  0 bad proto drop  0 if down drop
    0 input with dribble  0 input discard
    0 Rx pause
  TX
    34549861 unicast packets  1822766 multicast packets  2247607 broadcast packets
    38620234 output packets  38534039898 bytes
    17837212 jumbo packets    <========================  Jumbo packets Transmitted
    0 output errors  0 collision  0 deferred  0 late collision
    0 lost carrier  0 no carrier  0 babble
    0 Tx pause
  0 interface resets

TesoroDCRwBSwA#

However, issuing the 'show counters' command for the same port shows zero packets above 1518 bytes, in either direction:

TesoroDCRwBSwA# sh int eth113/1/5 counters detailed all
Ethernet113/1/5
  64 bit counters:
   0.                      rxHCTotalPkts = 20727315
   1.                       txHCTotalPks = 38628399
   2.                    rxHCUnicastPkts = 20684358
   3.                    txHCUnicastPkts = 34557774
   4.                  rxHCMulticastPkts = 23649
   5.                  txHCMulticastPkts = 1822882
   6.                  rxHCBroadcastPkts = 19308
   7.                  txHCBroadcastPkts = 2247743
   8.                         rxHCOctets = 8594691743
   9.                         txHCOctets = 38543398898
  10.                 rxTxHCPkts64Octets = 4742
  11.            rxTxHCpkts65to127Octets = 21245507
  12.           rxTxHCpkts128to255Octets = 3638512
  13.           rxTxHCpkts256to511Octets = 3034053
  14.          rxTxHCpkts512to1023Octets = 2411770
  15.         rxTxHCpkts1024to1518Octets = 8025526
  16.         rxTxHCpkts1519to1548Octets = 0   <========================  Jumbo packets Received or Transmitted
  17.                    rxHCTrunkFrames = 20727315
  18.                    txHCTrunkFrames = 38625433
  19.                     rxHCDropEvents = 0

  All Port Counters:
   0.                          InPackets = 20727315
   1.                           InOctets = 8594691743
   2.                        InUcastPkts = 20684358
   3.                        InMcastPkts = 23649
   4.                        InBcastPkts = 19308
   5.                        InJumboPkts = 3154161   <========================  Jumbo packets Received
   6.                  StormSuppressPkts = 0
   7.                         OutPackets = 38628399
   8.                          OutOctets = 38543398898
   9.                       OutUcastPkts = 34557774
  10.                       OutMcastPkts = 1822882
  11.                       OutBcastPkts = 2247743
  12.                       OutJumboPkts = 17841443   <========================  Jumbo packets Transmitted
  13.                   rxHCPkts64Octets = 323
  14.              rxHCPkts65to127Octets = 11821883
  15.             rxHCPkts128to255Octets = 1769680
  16.             rxHCPkts256to511Octets = 1931991
  17.            rxHCpkts512to1023Octets = 1275158
  18.           rxHCpkts1024to1518Octets = 774119
  19.           rxHCpkts1519to1548Octets = 0   <========================  Jumbo packets Received
  20.                   txHCPkts64Octets = 4419
  21.              txHCPkts65to127Octets = 9423624
  22.             txHCPkts128to255Octets = 1868832
  23.             txHCPkts256to511Octets = 1102062
  24.            txHCpkts512to1023Octets = 1136612
  25.           txHCpkts1024to1518Octets = 7251407
  26.           txHCpkts1519to1548Octets = 0   <========================  Jumbo packets Transmitted
  27.                        ShortFrames = 0
  28.                         Collisions = 0
  29.                          SingleCol = 0
  30.                           MultiCol = 0
  31.                            LateCol = 0
  32.                       ExcessiveCol = 0
  33.                        LostCarrier = 0
  34.                          NoCarrier = 0
  35.                              Runts = 0
  36.                             Giants = 0
  37.                           InErrors = 0
  38.                          OutErrors = 0
  39.                      InputDiscards = 0
  40.                      BadEtypeDrops = 0
  41.                        IfDownDrops = 0
  42.                    InUnknownProtos = 0
  43.                           txErrors = 0
  44.                              rxCRC = 0
  45.                             Symbol = 0
  46.                          txDropped = 0
  47.                      TrunkFramesTx = 38625433
  48.                      TrunkFramesRx = 20727315
  49.                         WrongEncap = 0
  50.                            Babbles = 0
  51.                          Watchdogs = 0
  52.                                ECC = 0
  53.                           Overruns = 0
  54.                          Underruns = 0
  55.                           Dribbles = 0
  56.                           Deferred = 0
  57.                            Jabbers = 0
  58.                           NoBuffer = 0
  59.                            Ignored = 0
  60.                        bpduOutLost = 0
  61.                        cos0OutLost = 0
  62.                        cos1OutLost = 0
  63.                        cos2OutLost = 0
  64.                        cos3OutLost = 0
  65.                        cos4OutLost = 0
  66.                        cos5OutLost = 0
  67.                        cos6OutLost = 0
  68.                        cos7OutLost = 0
  69.                            RxPause = 0
  70.                            TxPause = 0
  71.                             Resets = 0
  72.                            SQETest = 0
  73.                     InLayer3Routed = 0
  74.               InLayer3RoutedOctets = 0
  75.                    OutLayer3Routed = 0
  76.              OutLayer3RoutedOctets = 0
  77.                   OutLayer3Unicast = 0
  78.             OutLayer3UnicastOctets = 0
  79.                 OutLayer3Multicast = 0
  80.           OutLayer3MulticastOctets = 0
  81.                    InLayer3Unicast = 0
  82.              InLayer3UnicastOctets = 0
  83.                  InLayer3Multicast = 0
  84.            InLayer3MulticastOctets = 0
  85.              InLayer3AverageOctets = 0
  86.             InLayer3AveragePackets = 0
  87.             OutLayer3AverageOctets = 0
  88.            OutLayer3AveragePackets = 0

TesoroDCRwBSwA#

A check of the VMWare vswitch associated with this port indicates a setting of 1500 for MTU. 

Also, all Nexus 5020 ports are showing the presence of jumbo packets.  The Nexus 5020s connect to the Fabric Extenders, as well as each other (vPC), and to the upstream Nexus 7010s.  Interestingly, the Nexus 7010s do not show ANY jumbo packets.  Here are the interface counters for one Nexus 5020 and it's upstream Nexus 7010:

TesoroDCRwBSwA# sh cdp nei int eth 1/37
Capability Codes: R - Router, T - Trans-Bridge, B - Source-Route-Bridge
                  S - Switch, H - Host, I - IGMP, r - Repeater,
                  V - VoIP-Phone, D - Remotely-Managed-Device,
                  s - Supports-STP-Dispute

Device-ID             Local Intrfce Hldtme Capability  Platform      Port ID

TesoroDCCore1(TBM13301746)Eth1/37       174    R S I s   N7K-C7010     Eth1/17
TesoroDCRwBSwA#

TesoroDCRwBSwA# sh int eth 1/37
Ethernet1/37 is up
  Hardware: 10000 Ethernet, address: 000d.ecd4.062c (bia 000d.ecd4.062c)
  Description: To TesoroDCCore1 Eth 1/17
  MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA
  Port mode is trunk
  full-duplex, 10 Gb/s, media type is 10G
  Beacon is turned off
  Input flow-control is off, output flow-control is off
  Rate mode is dedicated
  Switchport monitor is off
  EtherType is 0x8100
  Last link flapped 3week(s) 5day(s)
  Last clearing of "show interface" counters never
  30 seconds input rate 2205440 bits/sec, 275680 bytes/sec, 813 packets/sec
  30 seconds output rate 2821256 bits/sec, 352657 bytes/sec, 687 packets/sec
  Load-Interval #2: 5 minute (300 seconds)
    input rate 2.02 Mbps, 570 pps; output rate 3.55 Mbps, 715 pps
  RX
    6508572753 unicast packets  92620006 multicast packets  122377981 broadcast packets
    6723570740 input packets  5536783708240 bytes
    2925141155 jumbo packets  0 storm suppression packets   <========================  Jumbo packets Received
    0 runts  0 giants  0 CRC  0 no buffer
    0 input error  0 short frame  0 overrun   0 underrun  0 ignored
    0 watchdog  0 bad etype drop  0 bad proto drop  0 if down drop
    0 input with dribble  0 input discard
    0 Rx pause
  TX
    79841117731 unicast packets  35023627 multicast packets  24503979 broadcast packets
    79900645337 output packets  61972162794577 bytes
    27090297260 jumbo packets   <========================  Jumbo packets Transmitted
    0 output errors  0 collision  0 deferred  0 late collision
    0 lost carrier  0 no carrier  0 babble
    0 Tx pause
  4 interface resets

TesoroDCRwBSwA#

TesoroDCCore1# sh int eth 1/17
Ethernet1/17 is up
  Hardware: 10000 Ethernet, address: 0026.0b91.614c (bia 0026.0b91.614c)
  Description: To TesoroDCRowBSwA Eth1/37
  MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec
  reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA
  Port mode is trunk
  full-duplex, 10 Gb/s, media type is 10G
  Beacon is turned off
  Auto-Negotiation is turned off
  Input flow-control is off, output flow-control is off
  Rate mode is shared
  Switchport monitor is off
  EtherType is 0x8100
  Last link flapped 3week(s) 5day(s)
  Last clearing of "show interface" counters never
  30 seconds input rate 4551368 bits/sec, 926 packets/sec
  30 seconds output rate 2250952 bits/sec, 723 packets/sec
  Load-Interval #2: 5 minute (300 seconds)
    input rate 3.83 Mbps, 773 pps; output rate 1.87 Mbps, 546 pps
  RX
    2739607515 unicast packets  3559663 multicast packets  9467871 broadcast packets
    2752635049 input packets  1319670987621 bytes
    0 jumbo packets  0 storm suppression packets   <========================  Jumbo packets Received
    0 runts  0 giants  0 CRC  0 no buffer
    0 input error  0 short frame  0 overrun   0 underrun  0 ignored
    0 watchdog  0 bad etype drop  0 bad proto drop  0 if down drop
    0 input with dribble  0 input discard
    0 Rx pause
  TX
    3555239664 unicast packets  9423339 multicast packets  21514304 broadcast packets
    3586177307 output packets  3155586707204 bytes
    0 jumbo packets   <========================  Jumbo packets Transmitted
    0 output error  0 collision  0 deferred  0 late collision
    0 lost carrier  0 no carrier  0 babble  0 output discard
    0 Tx pause
  3 interface resets

TesoroDCCore1#

Clearly there is a problem, as the N7010 says it did not transmit ANY jumbo packets, yet the connected N5020 says it received almost 3 Billion!

The N7010s are running ver. 5.1(3) of Nexus OS.

Could this be a software bug in Nexus OS on the 5020?  I didn't find any ddts about this issue.

Thank you in advance for your insight.

-rb

[Lucien Avramov]

You may have dot1q trunks and this increments the counter.

[ronbuchalski]

Lucien,

Yes, they are all dot1q trunks.  But why would that increment the jumbo packets counter?  The frames are still within the size range that is considered normal (not jumbo).

[Colby Beam]

Headers such as dot1q, lacp, etc can push packets over the size to be "considered" jumbo, thus increments this counter as Lucien said.

802.1Q does not actually encapsulate the original frame. Instead, for Ethernet frames, it adds a 32-bit field between the source MAC address and the EtherType/Length fields of the original frame, extending the minimum and maximum frame sizes from 64 and 1,518 bytes (octets) to 68 and 1,522 bytes. Two bytes are used for the tag protocol identifier (TPID), the other two bytes for tag control information (TCI).

[ronbuchalski]

Colby,

Yes, this is correct.  But, is this any reason why the Nexus 5K would count these full frames (1522 bytes) as jumbo frames?  I would think they would be considered maximum sized normal or standard frames.

The Catalyst 45xx and 65xx switches we have are switching the same types of frames, yet they don't count any jumbo frames.  Also, the Nexus 7Ks connected to the Nexus 5Ks are not incrementing their jumbo packet counters, and they are sending/receiving directly to Nexus 5k interfaces which ARE incrementing the jumbo packet counters.

I still say it looks like a software bug.

Thank you for your response.

-rb

[Colby Beam]

Can you paste the following here for the port going to the 7k

show hardware internal gatos port eX/Y counters rx

[diepes]

I have encounterd the same bug.

Lot of "jumbo" frames reported while there are none on the access ports.

Could this be the cause of "output errors" seen on Nexus 5K interface's ?

[stayd]

Hello ronbuchalski,

it is 5 years from your initial question, which is still without answer.

Today situation is pretty same like 5 years ago with new hardware and new software.

N7K (7.3(1)D1(1), the almost latest version ever today) is connected with N5548UP (7.3(1)N1(1)) at Eth3/1 and is connected with N5672UP-16G (7.3(1)N1(1)) at Eth3/3.

N5548UP is connected with N7K at E1/31.

N5672UP-16G is connected with N7K at E1/19.

On one side there are jumbo packets seen, on the N7K side there are not seen these packets. Miracle or bug or some sort of logical behavior ?

N7K:

Ethernet3/1 is up
admin state is up, Dedicated Interface
Hardware: 1000/10000 Ethernet, address: 84b5.17f4.WXYZ (bia 84b5.17f4.WXYZ)
Description: sw3
MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, medium is p2p
Port mode is FabricPath
full-duplex, 10 Gb/s, media type is 10G
Beacon is turned off
Auto-Negotiation is turned on
Input flow-control is off, output flow-control is off
Auto-mdix is turned on
Rate mode is dedicated
Switchport monitor is off
EtherType is 0x8100
EEE (efficient-ethernet) : n/a
Last link flapped 2d20h
Last clearing of "show interface" counters never
1 interface resets
Load-Interval #1: 30 seconds
30 seconds input rate 50640320 bits/sec, 6987 packets/sec
30 seconds output rate 65500920 bits/sec, 8930 packets/sec
input rate 50.64 Mbps, 6.99 Kpps; output rate 65.50 Mbps, 8.93 Kpps
Load-Interval #2: 5 minute (300 seconds)
300 seconds input rate 46732232 bits/sec, 6489 packets/sec
300 seconds output rate 57556760 bits/sec, 8182 packets/sec
input rate 46.73 Mbps, 6.49 Kpps; output rate 57.56 Mbps, 8.18 Kpps
RX
2808358848 unicast packets 134966 multicast packets 1332 broadcast packets
2808495135 input packets 3185458166948 bytes
0 jumbo packets 0 storm suppression packets<<<<<<<<<<<<<<<<NOTHING
0 runts 0 giants 0 CRC/FCS 0 no buffer
0 input error 0 short frame 0 overrun 0 underrun 0 ignored
0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop
0 input with dribble 435 input discard
0 Rx pause
TX
4030670513 unicast packets 22059060 multicast packets 24739062 broadcast p
ackets
4048814452 output packets 4952927480924 bytes
0 jumbo packets            <<<<<<<<<<<<<<<<NOTHING
0 output error 0 collision 0 deferred 0 late collision
0 lost carrier 0 no carrier 0 babble 0 output discard
0 Tx pause

Ethernet3/3 is up
admin state is up, Dedicated Interface
  Hardware: 1000/10000 Ethernet, address: 84b5.17f4.WXYZ (bia 84b5.17f4.WXYZ)
  Description: sw103
  MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec
  reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, medium is p2p
  Port mode is FabricPath
  full-duplex, 10 Gb/s, media type is 10G
  Beacon is turned off
  Auto-Negotiation is turned on
  Input flow-control is off, output flow-control is off
  Auto-mdix is turned on
  Rate mode is dedicated
  Switchport monitor is off
  EtherType is 0x8100
  EEE (efficient-ethernet) : n/a
  Last link flapped 2d20h
  Last clearing of "show interface" counters never
  1 interface resets
  Load-Interval #1: 30 seconds
    30 seconds input rate 20109696 bits/sec, 2078 packets/sec
    30 seconds output rate 5953128 bits/sec, 873 packets/sec
    input rate 20.11 Mbps, 2.08 Kpps; output rate 5.95 Mbps, 873 pps
  Load-Interval #2: 5 minute (300 seconds)
    300 seconds input rate 8402720 bits/sec, 952 packets/sec
    300 seconds output rate 5950288 bits/sec, 755 packets/sec
    input rate 8.40 Mbps, 952 pps; output rate 5.95 Mbps, 755 pps
  RX
    1891586937 unicast packets  126725 multicast packets  1 broadcast packets
    1891713664 input packets  2608851464919 bytes
    0 jumbo packets  0 storm suppression packets<<<<<<<<<<<<<<<<NOTHING
    0 runts  0 giants  0 CRC/FCS  0 no buffer
    0 input error  0 short frame  0 overrun   0 underrun  0 ignored
    0 watchdog  0 bad etype drop  0 bad proto drop  0 if down drop
    0 input with dribble  0 input discard
    0 Rx pause
  TX
    355181232 unicast packets  21774958 multicast packets  24756031 broadcast pa
ckets
    373042147 output packets  222096287040 bytes
    0 jumbo packets            <<<<<<<<<<<<<<<<NOTHING
    0 output error  0 collision  0 deferred  0 late collision
    0 lost carrier  0 no carrier  0 babble  0 output discard
    0 Tx pause

N5548UP:

Ethernet1/31 is up
Dedicated Interface

Hardware: 1000/10000 Ethernet, address: 547f.eec7.WXYZ (bia 547f.eec7.WXYZ)
Description: n7kA
MTU 1500 bytes, BW 10000000 Kbit,, BW 10000000 Kbit, DLY 10 usec
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, medium is p2p
Port mode is FabricPath
full-duplex, 10 Gb/s, media type is 10G
Beacon is turned off
Input flow-control is off, output flow-control is off
Rate mode is dedicated
Switchport monitor is off
EtherType is 0x8100
Last link flapped 2d20h
Last clearing of "show interface" counters never
9 interface resets
30 seconds input rate 43913240 bits/sec, 6364 packets/sec
30 seconds output rate 38453320 bits/sec, 5579 packets/sec
Load-Interval #2: 5 minute (300 seconds)
input rate 54.33 Mbps, 7.47 Kpps; output rate 42.03 Mbps, 6.01 Kpps
RX
205671343378 unicast packets 29525309491 multicast packets 772957577 broad
cast packets
235969610446 input packets 287098726010270 bytes
131089321291 jumbo packets 0 storm suppression bytes<<< SO SO MANY JOMBOS
0 runts 0 giants 0 CRC 0 no buffer
0 input error 0 short frame 0 overrun 0 underrun 0 ignored
0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop
0 input with dribble 5612844 input discard
0 Rx pause
TX
140834139822 unicast packets 9490530 multicast packets 1602 broadcast pack
ets
140843631954 output packets 144983984312409 bytes
80709855427 jumbo packets<<<<<<<<<<<<<< SO SO MANY JOMBOS
0 output error 0 collision 0 deferred 0 late collision
0 lost carrier 0 no carrier 0 babble 0 output discard
0 Tx pause

N5672UP-16G

Ethernet1/19 is up
Dedicated Interface

Hardware: 1000/10000 Ethernet, address: 00de.fb44.WXYZ (bia 00de.fb44.WXYZ)
MTU 1500 bytes, BW 10000000 Kbit,, BW 10000000 Kbit, DLY 10 usec
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, medium is p2p
Port mode is FabricPath
full-duplex, 10 Gb/s, media type is 10G
Beacon is turned off
Input flow-control is off, output flow-control is off
Rate mode is dedicated
Switchport monitor is off
EtherType is 0x8100
Last link flapped 2d20h
Last clearing of "show interface" counters never
3 interface resets
30 seconds input rate 3905424 bits/sec, 740 packets/sec
30 seconds output rate 1114952 bits/sec, 363 packets/sec
Load-Interval #2: 5 minute (300 seconds)
input rate 5.93 Mbps, 793 pps; output rate 7.17 Mbps, 860 pps
RX
11395244647 unicast packets 28852362727 multicast packets 619837791 broadc
ast packets
40867445165 input packets 33468037627440 bytes
9376588204 jumbo packets 0 storm suppression bytes<<<<< SO SO MANY JOMBOS
0 runts 0 giants 0 CRC 0 no buffer
0 input error 0 short frame 0 overrun 0 underrun 0 ignored
0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop
0 input with dribble 0 input discard
0 Rx pause
TX
31254416387 unicast packets 6265238 multicast packets 4 broadcast packets
31260681629 output packets 42694506884690 bytes
27003195526 jumbo packets<<<<<<<<<<<<<< SO SO MANY JOMBOS
0 output error 0 collision 0 deferred 0 late collision
0 lost carrier 0 no carrier 0 babble 0 output discard
0 Tx pause

[Jimmy Nguyen]

Ronbucholski,

I have exactly same issue.  Have you got correct answer for your original post.  Would new IOS version fix the issue?

Thank you for your post.

[mschrant]

I have the same scenario.  Has anyone learned the exact cause?

[stayd]

Me too, did you find the answer ?

[noemi.berry]

I know this post is 3 years old, but just in case anyone else is looking for an answer to the same problem:

The real question is: What size is "jumbo" ?  (it's not the MTU, that's *Maximum*).

The interface itself displays "MTU 1500" -- so why isn't the interface dropping your packets sized 1518?  Does MTU refer to the frame size after other headers are stripped off?

 Partial answer is that "show queueing interface" shows what the MTU is really set to, even though the interface reports 1500, but maybe 1500 is being used to define "input packet" and anything bigger is "jumbo."   It says "jumbo packet" though -- that implies that it includes a layer 3 header, unless the display isn't being fussy about "frame" meaning layer-2 and "packet" meaning layer-3.

Can anyone decode the confusing Nexus "show int" counters?

[dducamps01]

did you have a final reply to your question  ?

if yes I would be happy to get it

thanks 

[Steven Williams]

So is that the final answer to this? Trunk ports produce "jumbo" frames? Do we need to be configuring our switches for jumbo frames then?