Nexus 7k - port not working

[rjuchta]

Hi,

we have Nexus 7k with two N7K-SUP2 and one 1/10 Gbps Ethernet Module (N7K-F248XP-25), version 6.1(2)

The problem is that suddenly a port 1000base-SX on N7K-F248XP-25 changed state to down and we have no idea how to make it work again.

We changed SFP, tried "shut/ no shut" and "no switchport", tried to connect to a different switch. All without any results, the port is still down.

We didn't try resetting the module or upgrading the nxos yet ...

Did anybody face a similar issue ? Maybe there is something else we should perform ?

Thanks and happy New Year!

[mtsb]

Hi Ryszard,

Is the log showing anything error message before it went down? Please post the show interface ether x/y and show logging from the device. Is it only for that port or multiple ports having this issue?

Thanks,

Madhu

[rjuchta]

Hi Madhu,

the issue is with one port.

show interface:

Ethernet3/12 is down (Link not connected)

admin state is up, Dedicated Interface

  Hardware: 1000/10000 Ethernet, address: 8478.ac56.d8c2 (bia 4403.a718.ced3)

  Description: wroc1s63 via IPS

  Internet Address is 2.2.2.1/30

  MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec

  reliability 255/255, txload 1/255, rxload 1/255

  Encapsulation ARPA, medium is broadcast

  full-duplex, 1000 Mb/s, media type is 1G

  Beacon is turned off

  Auto-Negotiation is turned off

  Input flow-control is off, output flow-control is off

  Auto-mdix is turned off

  Rate mode is dedicated

  Switchport monitor is off

  EtherType is 0x8100

  EEE (efficient-ethernet) : n/a

  Last link flapped 1d23h

  Last clearing of "show interface" counters never

  4 interface resets

  30 seconds input rate 0 bits/sec, 0 packets/sec

  30 seconds output rate 0 bits/sec, 0 packets/sec

  Load-Interval #2: 5 minute (300 seconds)

    input rate 0 bps, 0 pps; output rate 0 bps, 0 pps

  RX

    119360 unicast packets  0 multicast packets  0 broadcast packets

    118760 input packets  168876720 bytes

    0 jumbo packets  0 storm suppression packets

    0 runts  0 giants  0 CRC  0 no buffer

    0 input error  0 short frame  0 overrun   0 underrun  0 ignored

    0 watchdog  0 bad etype drop  0 bad proto drop  0 if down drop

    0 input with dribble  0 input discard

    0 Rx pause

  TX

    298535 unicast packets  6873009 multicast packets  2244793 broadcast packets

    9415735 output packets  2393355700 bytes

    0 jumbo packets

    0 output error  0 collision  0 deferred  0 late collision

    0 lost carrier  0 no carrier  0 babble  0 output discard

    0 Tx pause

logs:

Dec 30 20:09:34 wroc1n71 : 2013 Dec 30 20:09:34 CET: %ETHPORT-5-IF_DOWN_CFG_CHANGE: Interface Ethernet3/12 is down(Config change) (after configuring port to "no switchport")

Dec 30 20:09:35 wroc1n71 : 2013 Dec 30 20:09:35 CET: %ETHPORT-5-SPEED: Interface Ethernet3/12, operational speed changed to 1 Gbps

Dec 30 20:09:35 wroc1n71 : 2013 Dec 30 20:09:35 CET: %ETHPORT-5-IF_DUPLEX: Interface Ethernet3/12, operational duplex mode changed to Full

Dec 30 20:09:35 wroc1n71 : 2013 Dec 30 20:09:35 CET: %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet3/12, operational Receive Flow Control state changed to off

Dec 30 20:09:35 wroc1n71 : 2013 Dec 30 20:09:35 CET: %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet3/12, operational Transmit Flow Control state changed to off

Dec 30 20:09:35 wroc1n71 : 2013 Dec 30 20:09:35 CET: %ETHPORT-5-IF_UP: Interface Ethernet3/12 is up in Layer3

Dec 30 20:09:35 wroc1n71 : 2013 Dec 30 20:09:35 CET: %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet3/12 is down (Link failure)

[mtsb]

Hi Ryszard,

I strongly suspect some layer 1 issue here. You have mentioned that you changed the SFP and tried to flap and change the configs as well which is strange. What does " sh inter eth 3/12 capabilities " and "show interface eth3/12 trans details" show for working and not working port? Do you see any difference in power levels between working and not working port? Bettter to open up a TAC case and get more details.

Thanks,

Madhu

[rjuchta]

Hi Madhu,

there is no differences for working and not-working port regarding capabilities:

  Model:                 N7K-F248XP-25

  Type (SFP capable):    1000base-SX

  Speed:                 1000,10000

  Duplex:                full

  Trunk encap. type:     802.1Q

  FabricPath capable:    yes

  Channel:               yes

  Broadcast suppression: percentage(0-100)

  Flowcontrol:           rx-(off/on/desired),tx-(off/on/desired)

  Rate mode:             dedicated

  Port mode:             Routed,Switched

  QOS scheduling:        rx-(8q4t),tx-(3p5q1t)

  CoS rewrite:           yes

  ToS rewrite:           yes

  SPAN:                  yes

  UDLD:                  yes

  MDIX:                  no

  TDR capable:           no

  Link Debounce:         yes

  Link Debounce Time:    yes

  FEX Fabric:            yes

  dot1Q-tunnel mode:     yes

  Pvlan Trunk capable:   yes

  Port Group Members:    9-12

  EEE (efficient-eth):   no

  PFC capable:           yes

Power levels for no-working and for working port:

Tx Power     -5.07 dBm     -5.10 dBm

Rx Power     -7.37 dBm     -6.07 dBm

Maybe this is a module hardware problem after all ...

[mtsb]

Hi Ryszard,

Could be a hardware issue. But unless we do a fault analysis we might not be able to confirm for sure. Get in touch with TAC for additional assistance.

Thanks,

Maduh

[Amit Singh]

Ok two things. First tell us the " show module " output and run som hardware disgnostics on the module.

Second thing is what is the device connected to the other end? Please try hard-coding the speed to 1G on the other end device. I have seen this kind of issue where you have to hard-code the speed to 1G on the networking devices to make it work specially in routers.

Try this and let us know.

Cheers,

-amit singh

[rjuchta]

Hi Amit,

show module command:

Mod  Ports  Module-Type                         Model              Status

---  -----  ----------------------------------- ------------------ ----------

3    48     1/10 Gbps Ethernet Module           N7K-F248XP-25      ok

Mod  Sw              Hw

---  --------------  ------

3    6.1(2)          1.0

Mod  MAC-Address(es)                         Serial-Num

---  --------------------------------------  ----------

3    44-03-a7-18-ce-c8 to 44-03-a7-18-ce-fb  JAF1649ASSS

Mod  Online Diag Status

---  ------------------

3    Pass

Chassis Ejector Support: Enabled

Ejector Status:

Left ejector CLOSE, Right ejector CLOSE, Module HW does support ejector based shutdown.

On the other end normally was connected IPS McAfee IntruShield, but we tried also connect to Catalyst 3550 (GBIC GE port).

What hardware diagnostics do you mean ? We cannot afford to do any disruptive tests.

thanks

[Amit Singh]

Please have a look at the below link, it should help you.

Yes, the diagnostics are disruptive but system will warn you for sure. Dont be in hurry to press enter :-)

http://www.cisco.com/en/US/docs/switches/datacenter/sw/best_practices/cli_mgmt_guide/hw_diag.html#wp1053607

Try to do the complete diagnostic test for specified port that is failing.

Cheers,

-amit singh

[rjuchta]

Hi Amit,

the link went up when I configured "speed nonegotiate" on the cat.3550 1G port.

Should I consider using this option as a good practice when connecting any device via 1G to Nexus ?

thanks