×

Warning message

  • Cisco Support Forums is in Read Only mode while the site is being migrated.
  • Cisco Support Forums is in Read Only mode while the site is being migrated.

Fiber refusing to pass layer 3 traffic / Copper trunk works

Unanswered Question
Dec 6th, 2013
User Badges:

Hello everyone



crazy issue here.. everything looks like it should be working but doesnt!


a summary would be our 3750 switches will not trunk over fiber (SM or MM) to our core 6500. They work just fine over copper.


The funny thing is, ONE 3750 works over a SM fiber run.



Drawing1.jpg


Our goal is to have redundancy to all switches, 1x copper trunk and 1x fiber trunk. running spanning tree mode rapid-pvst. VTP mode transparent (all VLANs are created manually on all switches --- they exist.)


however, the fiber trunks on 2 switches will only pass layer 2 (CDP neighbor has full detail, mac address-table builds off the 6500) but will not ping the directly connected 6500.  we are using cisco brand SFP/GBICs GLC-SX-MM &GLC-LH-SM  (we are sure the correct SFP is used with correct fiber type)


debug arp / debug ip packet shows Switch B & C never actually learn the core's mac address and tie it to the IP of 153.29.45.1. all switches have a default gateway of 153.29.45.1.


SWITCH A :


interface GigabitEthernet1/0/48

description CopperTrunk-to-Core

switchport trunk encapsulation dot1q

switchport trunk native vlan 48

switchport trunk allowed vlan 37,48,172

switchport mode trunk

interface GigabitEthernet1/1/1

description FiberTrunk-to-Core

switchport trunk encapsulation dot1q

switchport trunk native vlan 48

switchport trunk allowed vlan 37,48,172

switchport mode dynamic desirable

interface Vlan48

ip address 153.29.45.67 255.255.255.192

no ip redirects

no ip proxy-arp




SWITCH B:

interface GigabitEthernet1/1/1

description FiberTrunk-to-Core

switchport trunk encapsulation dot1q

switchport trunk native vlan 47

switchport trunk allowed vlan 37,47,172

switchport mode trunk

interface GigabitEthernet2/0/48

description CopperTrunk-to-Core

switchport trunk encapsulation dot1q

switchport trunk native vlan 47

switchport trunk allowed vlan 37,47,172

interface Vlan47

ip address 153.29.45.8 255.255.255.224

no ip redirects

no ip proxy-arp



SWITCH C:

interface GigabitEthernet1/0/48

description CopperTrunk-to-Core

switchport trunk encapsulation dot1q

switchport trunk native vlan 48

switchport trunk allowed vlan 37,48,172

switchport mode trunk

interface GigabitEthernet1/1/1

description FiberTrunk-to-Core

switchport trunk encapsulation dot1q

switchport trunk native vlan 48

switchport trunk allowed vlan 37,48,172

switchport mode trunk


interface Vlan48

ip address 153.29.45.81 255.255.255.192

no ip redirects

no ip proxy-arp



CORE(6500):


interface GigabitEthernet2/40

description Switch_B_Copper

switchport

switchport trunk encapsulation dot1q

switchport trunk native vlan 47

switchport trunk allowed vlan 37,47,172

switchport mode trunk

interface GigabitEthernet2/43

description Switch_A_Copper

switchport

switchport trunk encapsulation dot1q

switchport trunk native vlan 48

switchport trunk allowed vlan 37,48,172

switchport mode trunk

interface GigabitEthernet2/40

description Switch_C_Copper

switchport

switchport trunk encapsulation dot1q

switchport trunk native vlan 48

switchport trunk allowed vlan 37,48,172

switchport mode trunk

interface GigabitEthernet3/43

description Switch_A_Fiber

switchport

switchport trunk encapsulation dot1q

switchport trunk native vlan 48

switchport trunk allowed vlan 37,48,172

switchport mode trunk

interface GigabitEthernet3/40

description Switch_B_Fiber

switchport

switchport trunk encapsulation dot1q

switchport trunk native vlan 47

switchport trunk allowed vlan 36,37,47,172,500

switchport mode trunk

interface GigabitEthernet3/44

description Switch_C_Fiber

switchport

switchport trunk encapsulation dot1q

switchport trunk native vlan 48

switchport trunk allowed vlan 37,48,172

switchport mode trunk

interface Vlan47

description Internal_Management

ip address 153.29.45.1 255.255.255.224

no ip redirects

no ip proxy-arp

interface Vlan48

description Management

ip address 153.29.45.65 255.255.255.192



Summary: copper links work fine on all 3 switches. switch A & B refuse to pass layer 3 data on Fiber trunks. all were recently updated to version 12.2(53r) SE2 (c3740e-universalk9-mz.122-55.SE8.bin). Core is on 12.2 (33) SXI12 (s72033-adventerprisek9_wan-mz.122-33.SXI12.bin).



any suggestions are appreciated.

  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 0 (0 ratings)
Loading.
Marvin Rhoads Fri, 12/06/2013 - 07:07
User Badges:
  • Super Silver, 17500 points or more
  • Hall of Fame,
  • Cisco Designated VIP,

    2017 Firewalling, Network Management, VPN

While I don't normally set it, I notice that your switch A working fiber trunk has:


     switchport mode dynamic desirable


The non-working B and C fiber uplinks do not.


Have you tried using that command on them?

justin.barnhill Fri, 12/06/2013 - 07:20
User Badges:

i have. we have been troubleshooting this 24/7 for a couple days now, so that means things change constantly.


that being said, we traveled to the site containing switch A and plugged Switch C into it's fiber link. Switch C came up on layer 3. that means there is:


1. a problem with the SFPs (unlikely)

2. damaged fiber runs


we shall get our test equipment out tomorrow and being testing strands.


will update this thread with results, but thank you for the response Marvin!


have a great day.

Marvin Rhoads Fri, 12/06/2013 - 07:22
User Badges:
  • Super Silver, 17500 points or more
  • Hall of Fame,
  • Cisco Designated VIP,

    2017 Firewalling, Network Management, VPN

You're welcome.


You might troubleshoot the spans remotely using "show interface transceiver details". Also, turn on UDLD.

justin.barnhill Sun, 12/08/2013 - 22:12
User Badges:

so, we have localized the issue to the 6500.


Switch A is currently functioning on fiber port g3/43.


I created a test switch, and programmed the fiber port. i tried several ports on mod 3 (3/27, 3/42, 3/44, etc) as well as mod 4 (4/11). i even tried 6/1, the supervisor module's SFP port.


i then plugged the test switch into g3/43, and it worked. (good layer 3 ping)


we recently upgraded the 6500 from SXI6 to SXI12. i am looking over the release notes but not seeing any bugs like this.


i doubt a "hw-module module X reset" would do anything, since i tried 3 different modules.


thoughts??

justin.barnhill Sun, 12/08/2013 - 22:15
User Badges:

also the several ports i tried were all programmed exactly like 3/43, so it is not the config.

Leo Laohoo Sun, 12/08/2013 - 23:15
User Badges:
  • Super Gold, 25000 points or more
  • Hall of Fame,

    The Hall of Fame designation is a lifetime achievement award based on significant overall achievements in the community. 

  • Cisco Designated VIP,

    2017 LAN, Wireless

When you have fibre SFP in, post the output to the command "sh interface BLAH".

justin.barnhill Sun, 12/08/2013 - 23:39
User Badges:

Leo: here ya go



GigabitEthernet3/43 is up, line protocol is up (connected)

  Hardware is C6k 1000Mb 802.3, address is 2894.0f57.437a (bia 2894.0f57.437a)

  Description: FiberTrunk-to-SwitchA

  MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,

     reliability 255/255, txload 1/255, rxload 1/255

  Encapsulation ARPA, loopback not set

  Keepalive set (10 sec)

  Full-duplex, 1000Mb/s, media type is LH

  input flow-control is off, output flow-control is off

  Clock mode is auto

  ARP type: ARPA, ARP Timeout 04:00:00

  Last input 00:00:08, output 00:00:24, output hang never

  Last clearing of "show interface" counters never

  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0

  Queueing strategy: fifo

  Output queue: 0/40 (size/max)

  5 minute input rate 191000 bits/sec, 67 packets/sec

  5 minute output rate 752000 bits/sec, 112 packets/sec

     92584292 packets input, 25411211351 bytes, 0 no buffer

     Received 315527 broadcasts (83158 multicasts)

     0 runts, 0 giants, 0 throttles

     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

     0 watchdog, 0 multicast, 0 pause input

     0 input packets with dribble condition detected

     148923032 packets output, 114663742745 bytes, 0 underruns

     0 output errors, 0 collisions, 6 interface resets

     0 babbles, 0 late collision, 0 deferred

     0 lost carrier, 0 no carrier, 0 PAUSE output

     0 output buffer failures, 0 output buffers swapped out

GigabitEthernet6/1 is down, line protocol is down (notconnect)

  Hardware is C6k 1000Mb 802.3, address is 001a.a22d.6984 (bia 001a.a22d.6984)

  Description: FiberTestPort

  MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,

     reliability 255/255, txload 0/255, rxload 0/255

  Encapsulation ARPA, loopback not set

  Keepalive set (10 sec)

  Full-duplex, 1000Mb/s, media type is LH

  input flow-control is off, output flow-control is off

  Clock mode is auto

  ARP type: ARPA, ARP Timeout 04:00:00

  Last input 01:52:33, output 01:51:56, output hang never

  Last clearing of "show interface" counters never

  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0

  Queueing strategy: fifo

  Output queue: 0/40 (size/max)

  5 minute input rate 0 bits/sec, 0 packets/sec

  5 minute output rate 0 bits/sec, 0 packets/sec

     218 packets input, 26637 bytes, 0 no buffer

     Received 182 broadcasts (63 multicasts)

     0 runts, 0 giants, 0 throttles

     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

     0 watchdog, 0 multicast, 0 pause input

     0 input packets with dribble condition detected

     9476 packets output, 1084062 bytes, 0 underruns

     0 output errors, 0 collisions, 3 interface resets

     0 babbles, 0 late collision, 0 deferred

     0 lost carrier, 0 no carrier, 0 PAUSE output

     0 output buffer failures, 0 output buffers swapped out



let me know if you'd like to see anything else.

Leo Laohoo Sun, 12/08/2013 - 23:49
User Badges:
  • Super Gold, 25000 points or more
  • Hall of Fame,

    The Hall of Fame designation is a lifetime achievement award based on significant overall achievements in the community. 

  • Cisco Designated VIP,

    2017 LAN, Wireless

GigabitEthernet6/1 is down, line protocol is down (notconnect)

  Hardware is C6k 1000Mb 802.3, address is 001a.a22d.6984 (bia 001a.a22d.6984)

  Description: FiberTestPort

  MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,

     reliability 255/255, txload 0/255, rxload 0/255

Your SFP could be faulty.  I have NEVER seen a value like that before.


Can you try putting this SFP in a different port and different slot?

Leo Laohoo Sun, 12/08/2013 - 23:50
User Badges:
  • Super Gold, 25000 points or more
  • Hall of Fame,

    The Hall of Fame designation is a lifetime achievement award based on significant overall achievements in the community. 

  • Cisco Designated VIP,

    2017 LAN, Wireless

Wait .. there's another thing.  Post a picture of the SFP.  I want to see the front of the SFP where it will have the model number and the serial number. 

justin.barnhill Sun, 12/08/2013 - 23:46
User Badges:

there are several fiber ports in use on mod 3 and mod 4 working fine. rack switches, remote switches, etc.


any new trunks we try to add only get L2.




Mod Ports Card Type                                                       Model              Serial No.

--- ----- --------------------------------------                          ------------------ -----------

  1   48  CEF720 48 port 10/100/1000mb Ethernet  WS-X6748-GE-TX     SAD082702MJ

  2   48  CEF720 48 port 10/100/1000mb Ethernet  WS-X6748-GE-TX     SAL16042APK

  3   48  CEF720 48 port 1000mb SFP                        WS-X6748-SFP         SAL16063LT2

  4   48  CEF720 48 port 1000mb SFP                        WS-X6748-SFP         SAL16063LPT

  5    2  Supervisor Engine 720 (Hot)                         WS-SUP720-3B        SAL1127T8BG

  6    2  Supervisor Engine 720 (Active)                     WS-SUP720-3B        SAL1253F312


justin.barnhill Mon, 12/09/2013 - 01:33
User Badges:

Leo: you are seeing 0 tx/rx load because it was unplugged. here is the result when its plugged in:



GigabitEthernet6/1 is up, line protocol is up (connected)

  Hardware is C6k 1000Mb 802.3, address is 001a.a22d.6984 (bia 001a.a22d.6984)

  Description: FiberTestPort

  MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,

     reliability 255/255, txload 1/255, rxload 1/255

  Encapsulation ARPA, loopback not set

  Keepalive set (10 sec)

  Full-duplex, 1000Mb/s, media type is LH

  input flow-control is off, output flow-control is off

  Clock mode is auto

  ARP type: ARPA, ARP Timeout 04:00:00

  Last input 00:00:12, output 00:00:39, output hang never

  Last clearing of "show interface" counters never

  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0

  Queueing strategy: fifo

  Output queue: 0/40 (size/max)

  5 minute input rate 0 bits/sec, 0 packets/sec

  5 minute output rate 7000 bits/sec, 11 packets/sec

     439 packets input, 54271 bytes, 0 no buffer

     Received 390 broadcasts (131 multicasts)

     0 runts, 0 giants, 0 throttles

     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

     0 watchdog, 0 multicast, 0 pause input

     0 input packets with dribble condition detected

     20397 packets output, 2134848 bytes, 0 underruns

     0 output errors, 0 collisions, 3 interface resets

     0 babbles, 0 late collision, 0 deferred

     0 lost carrier, 0 no carrier, 0 PAUSE output

     0 output buffer failures, 0 output buffers swapped out




also here are the pictures of the SFPs from both 4/43 and 6/1.






Leo Laohoo Mon, 12/09/2013 - 01:45
User Badges:
  • Super Gold, 25000 points or more
  • Hall of Fame,

    The Hall of Fame designation is a lifetime achievement award based on significant overall achievements in the community. 

  • Cisco Designated VIP,

    2017 LAN, Wireless

Hmmmmm ... Thanks for the pix.  Not what I was hoping. 


Can you do the following:


1.  Plug one of the SFP into 5/1 and plug the other into 6/1.  Test and see if it works.

2.  Plug one SFP into another port other than 5 or 6.  Plug another into 5/1.  Test to see if this works. 

3.  Plug both SFP into 3750 and see if this works.


Note:  Do NOT enable the SFP to go active for very long or you will permanently burn your SFP.

justin.barnhill Mon, 12/09/2013 - 04:09
User Badges:

Leo, the issue is new 3750s will not pass layer 3 across any fiber trunk to the 6500. i have a test switch physically sitting in front of the 6500, with an LC-LC single-mode patch cable. it will not ping thru 5/1, 6/1, or any other port except 3/43 (existing port before the SXI12 upgrade) - all ports have exact same config as 3/43.

justin.barnhill Mon, 01/13/2014 - 01:45
User Badges:

so, after troubleshooting for 7 hours with a Cisco switching expert, we finally determined the cause. Turns out, 1 of 2 supervisor modules (WS-SUP720-3B) had fabric errors, resulting in fiber ports unable to build trunks/access.


replaced the supervisor module with a new one (RMA) and everything is back to normal.


posting this fix action in case others have the same issue.


- Justin

Actions

This Discussion

Related Content