cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
16534
Views
10
Helpful
7
Replies

Port down (Link failure) on FEX (2232TM)

Jozef Staruch
Level 1
Level 1

Hi All,

I would like to share with you one problem. We have FEX (2232TM) connected with 2 x NEXUS N5K-C5596UP.

One port  Ethernet 101/1/7 which was connected to Dell Server went down...

It was working for a couple of weeks at least. Then the link went down and now I'm unable to get anything connected to this port

As I said, this already happened before, but unfortunately I can't remember if it was on the same port/fex ... from the fex uptime it seems that this was on another fex

I was able to get the link back up by doing a power cycle of the fex

At the moment Dell server is connected to different port Ethernet101/1/4.. See config bellow:

. config, how ports are configured

   interface Ethernet101/1/7

   description VM008 Network 1

   switchport mode trunk

   switchport trunk allowed vlan 102,105,156,192,204

show interface ethernet 101/1/7 transceiver details

   debrscswi001# sho int e 101/1/7 trans det

   Ethernet101/1/7

   transceiver is not applicable

   (it's a 2232TM fex with 1/10 Gig copper ports only)

   interface Ethernet101/1/4

   description VM008 Netw 1 Tmp

   switchport mode trunk

   switchport trunk allowed vlan 102,105,156,192,204

2012 Dec 18 13:50:29 debrscswi001 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet101/1/7 is down (Link failure)

2012 Dec 18 13:50:33 debrscswi001 last message repeated 1 time

debrscswi001# show interface statu err-disabled

--------------------------------------------------------------------------------

Port         Name               Status   Reason

--------------------------------------------------------------------------------

Anbei noch ein paar Infos:

Hardware

cisco Nexus5596 Chassis ("O2 48X10GE/Modular Supervisor")

Intel(R) Xeon(R) CPU         with 8263848 kB of memory.

Processor Board ID FOC16270LF3

Device name: debrscswi001

Software

BIOS:     version 3.6.0

loader:   version N/A

kickstart: version 5.2(1)N1(1b)

system:   version 5.2(1)N1(1b)

-----------------------------------------------------------------------------

debrscswi001# sho fex 101 det

FEX: 101 Description: debrscfex101   state: Online

FEX version: 5.2(1)N1(1b) [Switch version: 5.2(1)N1(1b)]

FEX Interim version: 5.2(1)N1(1b)

Switch Interim version: 5.2(1)N1(1b)

Extender Serial: SSI153704R5

Extender Model: N2K-C2232TM-10GE, Part No: 73-13626-03

Card Id: 164, Mac Addr: f0:f7:55:93:a8:c2, Num Macs: 64

Module Sw Gen: 21 [Switch Sw Gen: 21]

post level: complete

Pinning-mode: static   Max-links: 1

Fabric port for control traffic: Eth3/16

FCoE Admin: false

FCoE Oper: true

FCoE FEX AA Configured: false

Fabric interface state:

   Po101 - Interface Up. State: Active

   Eth3/15 - Interface Up. State: Active

   Eth3/16 - Interface Up. State: Active

Fex Port       State Fabric Port

       Eth101/1/1   Up       Po101

       Eth101/1/2   Up       Po101

       Eth101/1/3   Up       Po101

       Eth101/1/4   Up       Po101

       Eth101/1/5   Up       Po101

       Eth101/1/6   Up       Po101

       Eth101/1/7 Down       Po101

       Eth101/1/8   Up       Po101

       Eth101/1/9   Up       Po101

     Eth101/1/10 Down       Po101

    Eth101/1/11   Up       Po101

     Eth101/1/12   Up       Po101

     Eth101/1/13 Down       Po101

     Eth101/1/14 Down       Po101

     Eth101/1/15 Down       Po101

     Eth101/1/16 Down       Po101

     Eth101/1/17 Down       Po101

     Eth101/1/18 Down       Po101

     Eth101/1/19 Down       Po101

     Eth101/1/20 Down       Po101

     Eth101/1/21 Down       Po101

     Eth101/1/22 Down       Po101

     Eth101/1/23 Down       Po101

     Eth101/1/24 Down       Po101

     Eth101/1/25 Down       Po101

     Eth101/1/26   Up       Po101

     Eth101/1/27   Up       Po101

     Eth101/1/28   Up       Po101

     Eth101/1/29   Up       Po101

     Eth101/1/30   Up       Po101

     Eth101/1/31   Up       Po101

     Eth101/1/32   Up       Po101

Logs:

10/03/2012 12:06:28.689867: Module register received

10/03/2012 12:06:28.690986: Registration response sent

10/03/2012 12:06:29.301434: Module Online Sequence

10/03/2012 12:06:33.748560: Module Online

----------------------------------------------------------------------

debrscswi001# show int e 101/1/7

Ethernet101/1/7 is down (Link not connected)

Hardware: 1000/10000 Ethernet, address: f0f7.5593.a8c8 (bia f0f7.5593.a8c8)

Description: VM008 Network 1

MTU 1500 bytes, BW 0 Kbit, DLY 10 usec

reliability 0/255, txload 1/255, rxload 1/255

Encapsulation ARPA

Port mode is trunk

auto-duplex, auto-speed

Beacon is turned off

Input flow-control is off, output flow-control is on

Switchport monitor is off

EtherType is 0x8100

Last link flapped 2week(s) 6day(s)

Last clearing of "show interface" counters 2w6d

30 seconds input rate 0 bits/sec, 0 packets/sec

30 seconds output rate 0 bits/sec, 0 packets/sec

Load-Interval #2: 5 minute (300 seconds)

   input rate 0 bps, 0 pps; output rate 0 bps, 0 pps

RX

   0 unicast packets 0 multicast packets 0 broadcast packets

   8 input packets 244 bytes

   0 jumbo packets 0 storm suppression bytes

   7 runts 0 giants 1 CRC 0 no buffer

   8 input error 0 short frame 0 overrun   0 underrun 0 ignored

   0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop

   0 input with dribble 0 input discard

   0 Rx pause

TX

   0 unicast packets 0 multicast packets 0 broadcast packets

   0 output packets 0 bytes

   0 jumbo packets

   0 output errors 0 collision 0 deferred 0 late collision

   0 lost carrier 0 no carrier 0 babble 0 output discard

   0 Tx pause

0 interface resets

Thank you for any advice...

7 Replies 7

Leo Laohoo
Hall of Fame
Hall of Fame
reliability 0/255, txload 1/255, rxload 1/255

Did you test the cable connected downstream to the FEX?  This looks like a Layer 1 issue.

Thank you for hint. Do you think it can be hardware problem as well?

The normal situation is always reliability 255/255

Hi Jozef,

leolaohoo is right.

Most likely the issue is layer-1 related. Log message also points to the layer-1 failure:

%ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet101/1/7 is down (Link failure)

Since the server now works fine on the other port - please check the cable/sfp module on the FEX - if their replacement won't fix the issue please open the TAC case.

HTH,

Alex

jessy5608
Level 1
Level 1

Hi Jozef,

Were you able to find a resoution to this issue that was not related to the cable/sfp module?  Was something on the switchport or switch misconfigured that caused the issue?  We are seeing the same problem in our enviroment.  Any help would be greatly appreciated.

-Jessy

Post the output of the relevant commands.

cisco answer I have been looking further into this and found the following bug, which has quite some similarities, although it is not an exact match: CSCuc51083 'Down' 2232 interfaces do not come 'Up' after ISSU to 5.2(1)n1(1b) => what is the same: - same NX-OS version with the problem - exact same type of FEX: N5K-C2232TM - is that also here only reload of the FEX solves this problem. - Trying to attach any other type of device to the affected port, doesn't help, not able to bring up any of these. What is different: is the trigger, in this bug an ISSU, while in your case the port was in use and then went down. In your case I saw last upgrade was: ----- reset reason for Supervisor-module 1 (from Supervisor in slot 1) --- 1) At 363117 usecs after Wed Oct  3 12:04:31 2012     Reason: Disruptive upgrade     Service:     Version: 5.2(1)N1(1) => knowing these similarities, it's quite possible that also some other sort of trigger exists other than an ISSU which could hit this defect, so I believe we indeed hit the same bug. Hence I would recommend on the long term to evaluate an upgrade to a fixed version of this bug. (short term fix is a reload of the FEX).

Jozef,

I will see if we have the same bug. Thank you for the quick response. 

-Jessy

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community:

Innovations in Cisco Full Stack Observability - A new webinar from Cisco