cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
72690
Views
15
Helpful
23
Replies

Interface getting down

Fernando Galvao
Level 1
Level 1
I'm having a Nexus 3064pq in Layer core being a core network. Today I have a problem where a 10Gbps port (1/36) is falling out of nowhere. It gets down and only returns when we remove the gbic cisco and put it back. We already changed the Gbic and the problem continues.

What can it be? Below the log:

show logging logfile | include 1/36
2017 Oct 12 14:59:00 UTC CORE-CISCO %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet1/36 is down (Transceiver Absent)
2017 Oct 12 14:59:04 UTC CORE-CISCO %ETHPORT-5-SPEED: Interface Ethernet1/36, operational speed changed to 10 Gbps
2017 Oct 12 14:59:04 UTC CORE-CISCO %ETHPORT-5-IF_DUPLEX: Interface Ethernet1/36, operational duplex mode changed to Full
2017 Oct 12 14:59:04 UTC CORE-CISCO %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet1/36, operational Receive Flow Control state changed to off
2017 Oct 12 14:59:04 UTC CORE-CISCO %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet1/36, operational Transmit Flow Control state changed to off
2017 Oct 12 14:59:04 UTC CORE-CISCO %ETHPORT-5-IF_UP: Interface Ethernet1/36 is up in mode trunk
2017 Oct 12 14:59:04 UTC CORE-CISCO %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/36 is down (Link failure)
2017 Oct 12 15:41:31 UTC CORE-CISCO %ETHPORT-5-IF_HARDWARE: Interface Ethernet1/36, hardware type changed to No-Transceiver
2017 Oct 12 15:42:10 UTC CORE-CISCO %ETHPORT-5-SPEED: Interface Ethernet1/36, operational speed changed to 10 Gbps
2017 Oct 12 15:42:10 UTC CORE-CISCO %ETHPORT-5-IF_DUPLEX: Interface Ethernet1/36, operational duplex mode changed to Full
2017 Oct 12 15:42:10 UTC CORE-CISCO %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet1/36, operational Receive Flow Control state changed to off
2017 Oct 12 15:42:10 UTC CORE-CISCO %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet1/36, operational Transmit Flow Control state changed to off
2017 Oct 12 15:42:10 UTC CORE-CISCO %ETHPORT-5-IF_UP: Interface Ethernet1/36 is up in mode trunk
2017 Oct 13 15:41:39 UTC CORE-CISCO %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/36 is down (Link failure)
2017 Oct 13 15:41:43 UTC CORE-CISCO %ETHPORT-5-SPEED: Interface Ethernet1/36, operational speed changed to 10 Gbps
2017 Oct 13 15:41:43 UTC CORE-CISCO %ETHPORT-5-IF_DUPLEX: Interface Ethernet1/36, operational duplex mode changed to Full
2017 Oct 13 15:41:43 UTC CORE-CISCO %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet1/36, operational Receive Flow Control state changed to off
2017 Oct 13 15:41:43 UTC CORE-CISCO %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet1/36, operational Transmit Flow Control state changed to off
2017 Oct 13 15:41:43 UTC CORE-CISCO %ETHPORT-5-IF_UP: Interface Ethernet1/36 is up in mode trunk
2017 Oct 17 15:57:57 UTC CORE-CISCO %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/36 is down (Link failure)
2017 Oct 17 15:58:00 UTC CORE-CISCO %ETHPORT-5-SPEED: Interface Ethernet1/36, operational speed changed to 10 Gbps
2017 Oct 17 15:58:00 UTC CORE-CISCO %ETHPORT-5-IF_DUPLEX: Interface Ethernet1/36, operational duplex mode changed to Full
2017 Oct 17 15:58:00 UTC CORE-CISCO %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet1/36, operational Receive Flow Control state changed to off
2017 Oct 17 15:58:00 UTC CORE-CISCO %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet1/36, operational Transmit Flow Control state changed to off
2017 Oct 17 15:58:00 UTC CORE-CISCO %ETHPORT-5-IF_UP: Interface Ethernet1/36 is up in mode trunk
2017 Oct 17 16:31:56 UTC CORE-CISCO %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/36 is down (Link failure)
2017 Oct 17 16:31:59 UTC CORE-CISCO %ETHPORT-5-SPEED: Interface Ethernet1/36, operational speed changed to 10 Gbps
2017 Oct 17 16:31:59 UTC CORE-CISCO %ETHPORT-5-IF_DUPLEX: Interface Ethernet1/36, operational duplex mode changed to Full
2017 Oct 17 16:31:59 UTC CORE-CISCO %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet1/36, operational Receive Flow Control state changed to off
2017 Oct 17 16:31:59 UTC CORE-CISCO %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet1/36, operational Transmit Flow Control state changed to off
2017 Oct 17 16:31:59 UTC CORE-CISCO %ETHPORT-5-IF_UP: Interface Ethernet1/36 is up in mode trunk
2017 Oct 17 17:00:57 UTC CORE-CISCO %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/36 is down (Link failure)
2017 Oct 17 17:10:11 UTC CORE-CISCO %ETHPORT-5-IF_HARDWARE: Interface Ethernet1/36, hardware type changed to No-Transceiver
2017 Oct 17 17:10:58 UTC CORE-CISCO %ETHPORT-5-SPEED: Interface Ethernet1/36, operational speed changed to 10 Gbps
2017 Oct 17 17:10:58 UTC CORE-CISCO %ETHPORT-5-IF_DUPLEX: Interface Ethernet1/36, operational duplex mode changed to Full
2017 Oct 17 17:10:58 UTC CORE-CISCO %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet1/36, operational Receive Flow Control state changed to off
2017 Oct 17 17:10:58 UTC CORE-CISCO %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet1/36, operational Transmit Flow Control state changed to off
2017 Oct 17 17:10:59 UTC CORE-CISCO %ETHPORT-5-IF_UP: Interface Ethernet1/36 is up in mode trunk
2017 Oct 17 17:12:08 UTC CORE-CISCO %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/36 is down (Link failure)
2017 Oct 17 17:12:11 UTC CORE-CISCO %ETHPORT-5-IF_HARDWARE: Interface Ethernet1/36, hardware type changed to No-Transceiver
2017 Oct 17 17:12:32 UTC CORE-CISCO %ETHPORT-5-SPEED: Interface Ethernet1/36, operational speed changed to 10 Gbps
2017 Oct 17 17:12:32 UTC CORE-CISCO %ETHPORT-5-IF_DUPLEX: Interface Ethernet1/36, operational duplex mode changed to Full
2017 Oct 17 17:12:32 UTC CORE-CISCO %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet1/36, operational Receive Flow Control state changed to off
2017 Oct 17 17:12:32 UTC CORE-CISCO %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet1/36, operational Transmit Flow Control state changed to off
2017 Oct 17 17:12:32 UTC CORE-CISCO %ETHPORT-5-IF_UP: Interface Ethernet1/36 is up in mode trunk
2017 Oct 20 15:26:53 UTC CORE-CISCO %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/36 is down (Link failure)
2017 Oct 20 15:38:24 UTC CORE-CISCO %ETHPORT-5-IF_HARDWARE: Interface Ethernet1/36, hardware type changed to No-Transceiver
2017 Oct 20 15:38:38 UTC CORE-CISCO %ETHPORT-5-SPEED: Interface Ethernet1/36, operational speed changed to 10 Gbps
2017 Oct 20 15:38:38 UTC CORE-CISCO %ETHPORT-5-IF_DUPLEX: Interface Ethernet1/36, operational duplex mode changed to Full
2017 Oct 20 15:38:38 UTC CORE-CISCO %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet1/36, operational Receive Flow Control state changed to off
2017 Oct 20 15:38:38 UTC CORE-CISCO %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet1/36, operational Transmit Flow Control state changed to off
2017 Oct 20 15:38:38 UTC CORE-CISCO %ETHPORT-5-IF_UP: Interface Ethernet1/36 is up in mode trunk
 

23 Replies 23

Reza Sharifi
Hall of Fame
Hall of Fame

If this is a fiber port, try replacing the cable and if that does not help replace the optic (sfp).

HTH

We already did that.

We already did that.

Hello,

 

check if 'show port internal info/event-history' gives you more information...

in what context is this commnand? 

Andrea Testino
Cisco Employee
Cisco Employee

Hi Fernando,

 

Could you let us know what NX-OS version this 3064 is running and what kind of device is at the other end of Eth1/36? Additionally, if you could share the outputs to the following, that would be helpful (text file attachment, these will be long):

show platform software ethpc event-history interface e1/36
show system internal ethpm event-history interface e1/36
show interface e1/36 transceiver details 

Thank you,

 

- Andrea

 

- Andrea, CCIE #56739 R&S

Hi,

 

 

The other end that connects to this switch is a router or dwdw from the carrier that provides us the link IX.br
As we move from port 36 today to 35 to see if it is resolved, I will send the transaiver command from port 1/35 that is running from today.

 

Software
BIOS: version 2.5.0
loader: version N/A
kickstart: version 6.0(2)U2(4) [build 6.0(2)U2(3.6)]
system: version 6.0(2)U2(4) [build 6.0(2)U2(3.6)]
Power Sequencer Firmware:
Module 1: version v4.1

 

CORE-CISCO# show interface e1/35 transceiver details
Ethernet1/35
transceiver is present
type is 10Gbase-LR
name is HG GENUINE
part number is MTRS-02X13-G
revision is 1.0
serial number is MB16350410883
nominal bitrate is 10300 MBit/sec
Link length supported for 9/125um fiber is 10 km
cisco id is --
cisco extended id number is 4

SFP Detail Diagnostics Information (internal calibration)
----------------------------------------------------------------------------
Current Alarms Warnings
Measurement High Low High Low
----------------------------------------------------------------------------
Temperature 25.45 C 80.00 C -10.00 C 70.00 C 0.00 C
Voltage 3.32 V 3.63 V 2.97 V 3.46 V 3.13 V
Current 29.25 mA 90.00 mA 2.00 mA 80.00 mA 3.00 mA
Tx Power -3.26 dBm 2.49 dBm -10.22 dBm 0.49 dBm -8.21 dBm
Rx Power -9.35 dBm 2.49 dBm -16.57 dBm 0.49 dBm -14.43 dBm
----------------------------------------------------------------------------
Note: ++ high-alarm; + high-warning; -- low-alarm; - low-warning

 

 

The rest is in the attached file.

Fernando,

 

Seems that the issue persists in Eth1/36 as well based on the event-histories you shared. Here's what the N3K logs when the port bounces:

 

Background information:
The debounce logic is written in NX-OS to detect when we lose signal from our directly connected host or device. When signal is lost, the Nexus will start a debounce timer which is 100ms by default to wait for the other end-device to signal us back, if we do not detect anything at this point we will take the link down assuming the connected port is down as well to avoid black holing traffic. This can be modified up to 5 seconds but it is against best practices to do so. This can be done at the interface configuration level with "link debounce time 5000”. In other words, based on the Nexus logs, it would appear that the issue lies on the other side of this connection which could also explain why the problem is still present in Eth1/36.

 

Sample output from the file you shared:

'show platform software ethpc event-history interface e1/36'
<snip>
33) Event IF_PCFG_CMD, len: 8, at 444263 usecs after Sat Oct 21 05:47:47 2017
     Processed port command PORT_CMD_DISABLE(3) status 0


34) Event IF_LINK_ST, len: 34, at 436138 usecs after Sat Oct 21 05:47:47 2017
     Link down sent to 0xfe000000 - Failure, Reason: Link down debounce timer stopped and link is down


35) Event IF_LINK_ST, len: 34, at 435893 usecs after Sat Oct 21 05:47:47 2017
     Link down sent to 0xfe000000 - Failure, Reason: Link down debounce timer started


36) Event IF_LINK_ST, len: 34, at 268110 usecs after Sat Oct 21 04:51:22 2017
     Link up sent to 0xfe000000 - Success, Reason: SUCCESS, Speed: 10000, Mdix mode: 3, Flow control mode: 0, Port duplex: 2
  <snip...another example...>


59) Event IF_LINK_ST, len: 34, at 554438 usecs after Sat Oct 21 04:50:12 2017
     Link down sent to 0xfe000000 - Failure, Reason: Link down debounce timer stopped and link is down


60) Event IF_LINK_ST, len: 34, at 552828 usecs after Sat Oct 21 04:50:12 2017
     Link down sent to 0xfe000000 - Failure, Reason: Link down debounce timer started


61) Event IF_LINK_ST, len: 34, at 916406 usecs after Sat Oct 21 04:45:54 2017
     Link up sent to 0xfe000000 - Success, Reason: SUCCESS, Speed: 10000, Mdix mode: 3, Flow control mode: 0, Port duplex: 2

I'd contact the carrier that owns the router/end-device and have them troubleshoot L1 at their end. Naturally, if there's a patch on your side, try a different patch panel port to be sure as well.

 

 

Hope that helps!

 

- Andrea

- Andrea, CCIE #56739 R&S

Many thanks for the quick return. Could the problem be at the other end of the carrier? Because just take out and put the tranceiver on our side again it works.

 

 

 

OK, but the strange thing is that it is no longer connected on port 1/36. It is connected to port 1/35. This was changed today at the time shown below:

 2017 Oct 21 05:47:47 UTC CORE-CISCO %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/36 is down (Link failure)
2017 Oct 21 05:48:23 UTC CORE-CISCO %ETHPORT-5-IF_HARDWARE: Interface Ethernet1/36, hardware type changed to No-Transceiver

Fernando,

 

Regardless of which port it is connected to, does it still flap? Moreover, when you say you fix it by re-inserting the SFP, what does then trigger the issue again? Based on the data you provided originally the link is flapping rather frequently so I'm a bit confused as to what you mean you can fix it.

 

Thanks!

- Andrea, CCIE #56739 R&S

Come on, the problem is that the door stays down and in the log appear messages from my first post. To get back to working, just remove and put the tranceiver on the same door ... everything comes back to work for a few days. This problem has already happened 3 times.

As an attempt to solve, we have already changed the cable, tranceiver and today we change doors. Now it's on 1/35. It is operational but too early to say that it is resolved.

Is it a little clearer now?

Many thanks for the quick return.

Fernando,

 

Got it! Let's monitor and if Eth1/35 goes down again get the same logs and we can take a second look.

 

Thank you,

 

- Andrea

- Andrea, CCIE #56739 R&S

Thank you very much.


A question, can you tell the difference between:
spanning-tree mode rapid-pvst
and
spanning-tree port type edge trunk

Today I'm spanning-tree mode with rapid-pvst on 1/35 and I do not know if the spanning-tree port type edge trunk
is the best option. I saw a person in the forum with the same error that used the spanning-tree port type edge trunk in the interface along with the commands:

no negotiate auto
speed 10000
duplex full

and solved it. Does spanning-tree port type trunk protect against loop? because I was with him and I changed after receiving two loop messages on interface 1/36:

2017 Oct 21 04:44:38 UTC CORE-CISCO% FWM-2-STM_LOOP_DETECT: Loops detected in the network for mac 4c5e.0c6a.1801 among ports Po6 and Eth1 / 36 on vlan 902 - Disabling dynamic learning notifications for a period betwee
n 120 and 240 seconds on vlan 902
2017 Oct 21 04:44:38 UTC CORE-CISCO% FWM-2-STM_LOOP_DETECT: Loops detected in the network for mac 4c5e.0c6a.1801 among ports Po6 and Eth1 / 36 on vlan 2425 - Disabling dynamic learning notifications for a period betwe
in 120 and 240 seconds on vlan 2425



You now have rapid-pvst on port 1/35.

Ethernet interface1 / 35
  description LINK-PTT-SP
  switchport mode trunk
  switchport trunk allowed vlan 902-903,2425

See how strange, I put a dell X4000 switch in the middle between the link being received by a gbic cisco SFP + and passing it to the nexus 3064 with the cisco cable ready model SFP-H10GB-ACU7M and so far the problem has not happened anymore.
Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community:

Review Cisco Networking products for a $25 gift card