10Gig (tengigabit) CRC Errors between N5K500 and 3750X

Unanswered Question
Jul 16th, 2012

Dear Forum,

We have an issue with a 10 Gig connection between a Nexus 5548 and a 3750X 10Gig. There is also a HSRP group running over this link with aggressive timeouts which recently behaves somewhat unstable. We noticed a bunch of CRC errors on this link on the 3750X side (about 300 per hour) and assume that these are related to the HSRP troubles once or twice a day.

Originally the connection was made with a twinax cable. We already changed to MMF fiber with SR SFP+'s. The 10Gig Uplink Module in the 3750x has been replaced already. All to no avail. Now it's up to replace the 3750 (receives CRC) or the Nexus Chassis (sending device)

I was wondering if there is some experience on the list what is the most probably cause and solution here.

Thanks in advance,

Michael.

I have this problem too.
0 votes
  • 1
  • 2
  • 3
  • 4
  • 5
Average Rating: 5 (1 ratings)
Leo Laohoo Mon, 07/16/2012 - 02:01

On the 3750X, can you post the output to the command "sh controll e ten " please?

Dr. Michael Weller Mon, 07/16/2012 - 02:44

Sure:

C3750#show controllers ethernet-controller t
C3750#show controllers ethernet-controller tenGigabitEthernet 3/1/1

     Transmit TenGigabitEthernet3/1/1         Receive
574708109169 Bytes                     518900683562 Bytes
   1439005510 Unicast frames              3807510591 Unicast frames
   1700275095 Multicast frames            2013035702 Multicast frames
     97077957 Broadcast frames             361871415 Broadcast frames
            0 Too old frames            750584488179 Unicast bytes
            0 Deferred frames           149720142824 Multicast bytes
            0 MTU exceeded frames        40440767697 Broadcast bytes
            0 1 collision frames                   0 Alignment errors
            0 2 collision frames             1299355 FCS errors
            0 3 collision frames                   1 Oversize frames
            0 4 collision frames                   0 Undersize frames
            0 5 collision frames                   0 Collision fragments
            0 6 collision frames
            0 7 collision frames                 981 Minimum size frames
            0 8 collision frames          2689693081 65 to 127 byte frames
            0 9 collision frames          1727005063 128 to 255 byte frames
            0 10 collision frames         1091861080 256 to 511 byte frames
            0 11 collision frames         1515886754 512 to 1023 byte frames
            0 12 collision frames          497582417 1024 to 1518 byte frames
            0 13 collision frames                  0 Overrun frames
            0 14 collision frames                  0 Pause frames
            0 15 collision frames
            0 Excessive collisions                 2 Symbol error frames
            0 Late collisions                      4 Invalid frames, too large
            0 VLAN discard frames         2956654982 Valid frames, too large
            0 Excess defer frames                  0 Invalid frames, too small
     10050135 64 byte frames                       0 Valid frames, too small
   1897954866 127 byte frames
   2274249521 255 byte frames                      0 Too old frames
   1808046421 511 byte frames                      0 Valid oversize frames
   4247747354 1023 byte frames                     0 System FCS error frames
   1134905192 1518 byte frames                     0 RxPortFifoFull drop frame
    453339665 Too large frames
            0 Good (1 coll) frames
            0 Good (>1 coll) frames

C3750#show controllers ethernet-controller tenGigabitEthernet 4/1/1

     Transmit TenGigabitEthernet4/1/1         Receive
401013315009 Bytes                     487542547741 Bytes
   2315452233 Unicast frames              2487385055 Unicast frames
    315842017 Multicast frames            1505902339 Multicast frames
     59648917 Broadcast frames             185748001 Broadcast frames
            0 Too old frames            119286086552 Unicast bytes
            0 Deferred frames           103246245303 Multicast bytes
            0 MTU exceeded frames        24880634311 Broadcast bytes
            0 1 collision frames                   0 Alignment errors
            0 2 collision frames              983530 FCS errors
            0 3 collision frames                   0 Oversize frames
            0 4 collision frames                   0 Undersize frames
            0 5 collision frames                   0 Collision fragments
            0 6 collision frames
            0 7 collision frames               49004 Minimum size frames
            0 8 collision frames           445407751 65 to 127 byte frames
            0 9 collision frames          2540569644 128 to 255 byte frames
            0 10 collision frames          799266760 256 to 511 byte frames
            0 11 collision frames         2422515083 512 to 1023 byte frames
            0 12 collision frames           68360393 1024 to 1518 byte frames
            0 13 collision frames                  0 Overrun frames
            0 14 collision frames                  0 Pause frames
            0 15 collision frames
            0 Excessive collisions                 0 Symbol error frames
            0 Late collisions                      0 Invalid frames, too large
            0 VLAN discard frames         2198817586 Valid frames, too large
            0 Excess defer frames                  0 Invalid frames, too small
      4432313 64 byte frames                       0 Valid frames, too small
   2439601404 127 byte frames
   3641955592 255 byte frames                      0 Too old frames
   3604137484 511 byte frames                      0 Valid oversize frames
   2478595942 1023 byte frames                     0 System FCS error frames
   1831032940 1518 byte frames                     0 RxPortFifoFull drop frame
   1576089380 Too large frames
            0 Good (1 coll) frames
            0 Good (>1 coll) frames

I'm also concerned about the too large frames. It seems to me the Nexus uses Jumbo-Frames (don't know if it should, might be a default setting).

Oh, BTW.. I just saw the N5k is not a 5548, but a 5020. Sorry.

Leo Laohoo Mon, 07/16/2012 - 02:50

I'm very concerned about the high amount of FCS (Frame Check Sequence) line errors in both interfaces.

Forgot to ask the corresponding output to the commands "sh interface ten 3/1/1" and "sh interface ten 4/1/1".

Dr. Michael Weller Mon, 07/16/2012 - 03:00

MTU is 1500 bytes, on both sides (I think) see output below. Strangely enough the NX has a non-zero Jumboframe counter (increasing).

C3750#sh int te3/1/1
TenGigabitEthernet3/1/1 is up, line protocol is up (connected)
  Hardware is Ten Gigabit Ethernet, address is 4055.3992.111d (bia 4055.3992.111d)
  Description: TO_100C_PO913
  MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec,
     reliability 255/255, txload 2/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive not set
  Full-duplex, 10Gb/s, link type is auto, media type is SFP-10GBase-SR
  Media-type configured as  connector
  input flow-control is off, output flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:55, output 00:00:41, output hang never
  Last clearing of "show interface" counters 04:10:35
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 4011000 bits/sec, 1091 packets/sec
  5 minute output rate 110431000 bits/sec, 16489 packets/sec
     37040147 packets input, 35868731879 bytes, 0 no buffer
     Received 1071851 broadcasts (935322 multicasts)
     0 runts, 0 giants, 0 throttles
     1002 input errors, 1002 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 935322 multicast, 0 pause input
     0 input packets with dribble condition detected
     182446795 packets output, 69097435882 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     0 output buffer failures, 0 output buffers swapped out
C3750#sh int te4/1/1
TenGigabitEthernet4/1/1 is up, line protocol is up (connected)
  Hardware is Ten Gigabit Ethernet, address is 4055.39bd.001d (bia 4055.39bd.001d)
  Description: TO_100D_PO913
  MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive not set
  Full-duplex, 10Gb/s, link type is auto, media type is SFP-10GBase-SR
  Media-type configured as  connector
  input flow-control is off, output flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:14, output 00:00:33, output hang never
  Last clearing of "show interface" counters 04:10:38
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 161000 bits/sec, 215 packets/sec
  5 minute output rate 29493000 bits/sec, 9580 packets/sec
     3264201 packets input, 318586322 bytes, 0 no buffer
     Received 1130048 broadcasts (1020339 multicasts)
     0 runts, 0 giants, 0 throttles
     501 input errors, 501 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 1020339 multicast, 0 pause input
     0 input packets with dribble condition detected
     168673910 packets output, 87266013753 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     0 output buffer failures, 0 output buffers swapped out

N5K connected to 3/1/1:

Ethernet1/1 is up
  Hardware: 1000/10000 Ethernet, address: 0005.9b1f.8648 (bia 0005.9b1f.8648)
  Description: CHANNEL_TO_RO015A
  MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec
  reliability 255/255, txload 1/255, rxload 3/255
  Encapsulation ARPA
  Port mode is trunk
  full-duplex, 10 Gb/s, media type is 10G
  Beacon is turned off
  Input flow-control is off, output flow-control is off
  Rate mode is dedicated
  Switchport monitor is off
  EtherType is 0x8100
  Last link flapped 2d04h
  Last clearing of "show interface" counters never
  30 seconds input rate 96695496 bits/sec, 16464 packets/sec
  30 seconds output rate 4715024 bits/sec, 1868 packets/sec
  Load-Interval #2: 5 minute (300 seconds)
    input rate 148.33 Mbps, 20.23 Kpps; output rate 2.90 Mbps, 1.13 Kpps
  RX
    26173471641 unicast packets  6446061938 multicast packets  24295222 broadcas
t packets
    32643828801 input packets  16853116460189 bytes
    7772805495 jumbo packets  0 storm suppression packets
    0 runts  0 giants  0 CRC  0 no buffer
    0 input error  0 short frame  0 overrun   0 underrun  0 ignored
    0 watchdog  0 bad etype drop  0 bad proto drop  0 if down drop
    0 input with dribble  0 input discard
    0 Rx pause
  TX
    48369101239 unicast packets  462061134 multicast packets  81589744 broadcast
packets
    48912752123 output packets  56533126796163 bytes
    35421478216 jumbo packets
    6 output errors  0 collision  0 deferred  0 late collision
    0 lost carrier  0 no carrier  0 babble 0 output discard
    0 Tx pause
  3 interface resets

N5K connected to 4/1/1:
Ethernet1/1 is up
  Hardware: 1000/10000 Ethernet, address: 0005.9b1f.8188 (bia 0005.9b1f.8188)
  Description: CHANNEL_TO_RO0015A
  MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec
  reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA
  Port mode is trunk
  full-duplex, 10 Gb/s, media type is 10G
  Beacon is turned off
  Input flow-control is off, output flow-control is off
  Rate mode is dedicated
  Switchport monitor is off
  EtherType is 0x8100
  Last link flapped 2d04h
  Last clearing of "show interface" counters never
  30 seconds input rate 20643816 bits/sec, 10541 packets/sec
  30 seconds output rate 142592 bits/sec, 180 packets/sec
  Load-Interval #2: 5 minute (300 seconds)
    input rate 25.50 Mbps, 9.49 Kpps; output rate 159.20 Kbps, 142 pps
  RX
    119855915845 unicast packets  6017690609 multicast packets  20318382 broadca
st packets
    125893924837 input packets  124351081100162 bytes
    70763564145 jumbo packets  0 storm suppression packets
    0 runts  0 giants  1 CRC  0 no buffer
    1 input error  0 short frame  0 overrun   0 underrun  0 ignored
    0 watchdog  0 bad etype drop  0 bad proto drop  0 if down drop
    0 input with dribble  0 input discard
    0 Rx pause
  TX
    20958491666 unicast packets  503446052 multicast packets  52796318 broadcast
packets
    21514734036 output packets  29748981423583 bytes
    18964491576 jumbo packets
    0 output errors  0 collision  0 deferred  0 late collision
    0 lost carrier  0 no carrier  0 babble 0 output discard
    0 Tx pause
  3 interface resets


Leo Laohoo Mon, 07/16/2012 - 03:09

Ok, no output drops on the 3750X.  That's good news.

Out of curiousity, what kind of fibre optic cable are you using?  Are you using OM3 or OM4?

Dr. Michael Weller Mon, 07/16/2012 - 03:18

That's only OM3.. but only a very short cable, about 5m. Note that it used to be a direct attached SFP-H10GB-CU5M before.

Then, when the FCS errors were noticed, it was already replaced with this fibre connection last weekend. Hence I'd exclude an error in the cables or sfps since they've already been replaced with a completely different technology.

Those two lines are btw channeled with LACP, it's a multichassis VPC on the Nexus side.

Leo Laohoo Mon, 07/16/2012 - 03:36

Hmmm ... 5 meter cable? Is this a SINGLE piece? Meaning this fibre optic doesn't go to a FOBOT?

Judging from the first output, the problem is coming from the Nexus side.

One more thing: both Ten3/1/1 & Ten4/1/1 are coming from the same 5020?

If the answer is yes, then i think it's coming from the appliance.

ASIC: what ports in the 5020? Can you move one link to another port group?

Sent from Cisco Technical Support iPad App

Dr. Michael Weller Mon, 07/16/2012 - 03:47

Yes, one single piece of fibre, no fobot. The fibres have been brought in place instead of the SFP-H10GB using onsite spare parts, for testing if there's a problem.

No it's not the same 5020. It's a vpc pair. Eth1/1 on each chassis, see show int output above. What confuses me here is, that these are two N5K chassis and both produce CRC failures. Likewise, connection ends up on two different 3750chassis (combined in a stack), each reporting CRC failures. There is no single port ASIC affected. Do you know by heart how the ports are grouped on asics? I'd need to search for this info. But agains, ports are distributed over chassis.

Of course, the 5020 are in a vpc cluster and the 3750x are stacked. So errors might somehow spread over the systems.

I'm also confused the 5020s report incoming jumbo frames though the system mtu on the 3750 is 1500, least to talk about the interface mtu.

Leo Laohoo Mon, 07/16/2012 - 22:39

Ok.  You got me stump.

The FCS counters from the 3750X means the errors are coming from the 5K.

By the way, does the serial number(s) of the SFP+ start with an "A"?

Dr. Michael Weller Tue, 07/17/2012 - 01:57

No Problem,

since I wasn't sure which device to replace by RMA I opened a TAC case. The Tac is currently confused too. They deep analysed the Nexus w/o finding a problem and are now analysing the 3750, maybe looking for a software issue too.

The main point about this thing is that there is no common denominator.. TwinAx or Fibre, two 5020, two 3750. And still two ports showing the same effect.

SNs of the SFP-10G-SR begin with FNS btw. They should be ok and original, no OEM or similar.

If a solution is found, I'll post a note.

Leo Laohoo Tue, 07/17/2012 - 03:19

FNS means its made by Finnisar. It should be OK.

Normally, if I had a case like this, I'd replace both SFP+ and fibre optic cable three time.

But since it's happening to two different machines I'm no longer sure this is the case.

What IOS is the 3750X running?

Sent from Cisco Technical Support iPad App

Dr. Michael Weller Tue, 07/17/2012 - 03:57

> But since it's happening to two different machines I'm no longer sure this is the case.

And.. the crc had been detected first with twinax cables. The SFP+ are already a replacement for the Twinax.

12.2(55)SE1; IP-Services.  Actually it's a mixed stack: 2 3750X and 2 3750G-12S. I'm wondering if there is an issue with the current stack master.

Leo Laohoo Tue, 07/17/2012 - 14:45
12.2(55)SE1; IP-Services.  Actually it's a mixed stack: 2 3750X and 2 3750G-12S. I'm wondering if there is an issue with the current stack master.

Uh-oh.

I'm no big fan of this IOS level.  Try 12.2(55)SE4 or 12.2(55)SE5 if you can get away with upgrading the IOS.

Cory Peterson Wed, 08/08/2012 - 05:58

Hello,

We have noticed the exact same behaviour between our 3750x and Nexus 7k boxes running on Twinax. With the same rate of CRC errors.

Has TAC Been able to idntifiy this issue at all? Did you try to update the IOS?

We are running 12.2(58)SE2 on the 3750x.

Thanks!

Cory

Dr. Michael Weller Wed, 08/08/2012 - 08:10

Hello Cory, hello list..

It's good that you remind me of this thread, so I can actually post the solution. It's not directly related to the 3750 at all.

TAC analysed a wireshark capture and noticed a bunch of DTP packets with vlan tags.

DTP packets are not allowed to have vlan tags, so the 3750 complains about this framing error. Due to the asic and IOS design of the 3750, those errors are actually counted as CRC errors, even though it's not strictly a CRC error. The 3750 also drops these malformed packets.

The misleading behaviour of the 3750 is already known and documented:

http://tools.cisco.com/Support/BugToolKit/search/getBugDetails.do?method=fetchBugDetails&bugId=CSCdz33708

This leads us to the question where these packets originated:

Through a chain of Nexus Switches (5k and 7k) a few other IOS Switches have been connected to the network.

The nexus does not support DTP or dynamic trunking configuration. The Nexus Port was configured as switchport mode access within some vlan.

The IOS Switch had its port more or less in default configuration, most important: switchport mode dynamic. This switch now sends out DTP packets every now and then to possibly negotiate a trunk. The Nexus does not understand DTP and forwards these packets as ordinary packets in the access-vlan. All Nexus in the chain just forward these packets like any other data frame. Then they are forwarded through a trunk to the DTP aware 3750...

As a solution, 'switchport mode access' was configured in the switches connected to the access ports of the N7K. You should search for similar ports in your setups. Nexus/newer IOS Switches tend to complain about mismatching VLANs in CDP even when two access-ports are connected. This might be a hint to you, at which ports to look. Finally, you could make a packet capture on the uplink, looking for the tagged DTP frames and search for the source mac in your network.

I'm sure this will resolve your issue.

Regards,

Michael.

markwaltersccie Thu, 02/13/2014 - 15:22

Hi All -

It's my understanding that setting the port to "switchport mode access" does not actually stop DTP advertisements - it likely just removed the VLAN tag in this scenario.  "switchport nonegotiate" ceases DTP advertisements altogether.

Cheers

Mark

Actions

Login or Register to take actions

This Discussion

Posted July 16, 2012 at 1:52 AM
Stats:
Replies:16 Avg. Rating:5
Views:8177 Votes:0
Shares:0

Related Content

Discussions Leaderboard