01-09-2013 12:53 AM - edited 03-07-2019 10:59 AM
Hi All,
I would like to share with you one problem. We have FEX (2232TM) connected with 2 x NEXUS N5K-C5596UP.
One port Ethernet 101/1/7 which was connected to Dell Server went down...
It was working for a couple of weeks at least. Then the link went down and now I'm unable to get anything connected to this port
As I said, this already happened before, but unfortunately I can't remember if it was on the same port/fex ... from the fex uptime it seems that this was on another fex
I was able to get the link back up by doing a power cycle of the fex
At the moment Dell server is connected to different port Ethernet101/1/4.. See config bellow:
. config, how ports are configured
interface Ethernet101/1/7
description VM008 Network 1
switchport mode trunk
switchport trunk allowed vlan 102,105,156,192,204
show interface ethernet 101/1/7 transceiver details
debrscswi001# sho int e 101/1/7 trans det
Ethernet101/1/7
transceiver is not applicable
(it's a 2232TM fex with 1/10 Gig copper ports only)
interface Ethernet101/1/4
description VM008 Netw 1 Tmp
switchport mode trunk
switchport trunk allowed vlan 102,105,156,192,204
2012 Dec 18 13:50:29 debrscswi001 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet101/1/7 is down (Link failure)
2012 Dec 18 13:50:33 debrscswi001 last message repeated 1 time
debrscswi001# show interface statu err-disabled
--------------------------------------------------------------------------------
Port Name Status Reason
--------------------------------------------------------------------------------
Anbei noch ein paar Infos:
Hardware
cisco Nexus5596 Chassis ("O2 48X10GE/Modular Supervisor")
Intel(R) Xeon(R) CPU with 8263848 kB of memory.
Processor Board ID FOC16270LF3
Device name: debrscswi001
Software
BIOS: version 3.6.0
loader: version N/A
kickstart: version 5.2(1)N1(1b)
system: version 5.2(1)N1(1b)
-----------------------------------------------------------------------------
debrscswi001# sho fex 101 det
FEX: 101 Description: debrscfex101 state: Online
FEX version: 5.2(1)N1(1b) [Switch version: 5.2(1)N1(1b)]
FEX Interim version: 5.2(1)N1(1b)
Switch Interim version: 5.2(1)N1(1b)
Extender Serial: SSI153704R5
Extender Model: N2K-C2232TM-10GE, Part No: 73-13626-03
Card Id: 164, Mac Addr: f0:f7:55:93:a8:c2, Num Macs: 64
Module Sw Gen: 21 [Switch Sw Gen: 21]
post level: complete
Pinning-mode: static Max-links: 1
Fabric port for control traffic: Eth3/16
FCoE Admin: false
FCoE Oper: true
FCoE FEX AA Configured: false
Fabric interface state:
Po101 - Interface Up. State: Active
Eth3/15 - Interface Up. State: Active
Eth3/16 - Interface Up. State: Active
Fex Port State Fabric Port
Eth101/1/1 Up Po101
Eth101/1/2 Up Po101
Eth101/1/3 Up Po101
Eth101/1/4 Up Po101
Eth101/1/5 Up Po101
Eth101/1/6 Up Po101
Eth101/1/7 Down Po101
Eth101/1/8 Up Po101
Eth101/1/9 Up Po101
Eth101/1/10 Down Po101
Eth101/1/11 Up Po101
Eth101/1/12 Up Po101
Eth101/1/13 Down Po101
Eth101/1/14 Down Po101
Eth101/1/15 Down Po101
Eth101/1/16 Down Po101
Eth101/1/17 Down Po101
Eth101/1/18 Down Po101
Eth101/1/19 Down Po101
Eth101/1/20 Down Po101
Eth101/1/21 Down Po101
Eth101/1/22 Down Po101
Eth101/1/23 Down Po101
Eth101/1/24 Down Po101
Eth101/1/25 Down Po101
Eth101/1/26 Up Po101
Eth101/1/27 Up Po101
Eth101/1/28 Up Po101
Eth101/1/29 Up Po101
Eth101/1/30 Up Po101
Eth101/1/31 Up Po101
Eth101/1/32 Up Po101
Logs:
10/03/2012 12:06:28.689867: Module register received
10/03/2012 12:06:28.690986: Registration response sent
10/03/2012 12:06:29.301434: Module Online Sequence
10/03/2012 12:06:33.748560: Module Online
----------------------------------------------------------------------
debrscswi001# show int e 101/1/7
Ethernet101/1/7 is down (Link not connected)
Hardware: 1000/10000 Ethernet, address: f0f7.5593.a8c8 (bia f0f7.5593.a8c8)
Description: VM008 Network 1
MTU 1500 bytes, BW 0 Kbit, DLY 10 usec
reliability 0/255, txload 1/255, rxload 1/255
Encapsulation ARPA
Port mode is trunk
auto-duplex, auto-speed
Beacon is turned off
Input flow-control is off, output flow-control is on
Switchport monitor is off
EtherType is 0x8100
Last link flapped 2week(s) 6day(s)
Last clearing of "show interface" counters 2w6d
30 seconds input rate 0 bits/sec, 0 packets/sec
30 seconds output rate 0 bits/sec, 0 packets/sec
Load-Interval #2: 5 minute (300 seconds)
input rate 0 bps, 0 pps; output rate 0 bps, 0 pps
RX
0 unicast packets 0 multicast packets 0 broadcast packets
8 input packets 244 bytes
0 jumbo packets 0 storm suppression bytes
7 runts 0 giants 1 CRC 0 no buffer
8 input error 0 short frame 0 overrun 0 underrun 0 ignored
0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop
0 input with dribble 0 input discard
0 Rx pause
TX
0 unicast packets 0 multicast packets 0 broadcast packets
0 output packets 0 bytes
0 jumbo packets
0 output errors 0 collision 0 deferred 0 late collision
0 lost carrier 0 no carrier 0 babble 0 output discard
0 Tx pause
0 interface resets
Thank you for any advice...
01-09-2013 01:01 AM
reliability 0/255, txload 1/255, rxload 1/255
Did you test the cable connected downstream to the FEX? This looks like a Layer 1 issue.
01-09-2013 01:29 AM
Thank you for hint. Do you think it can be hardware problem as well?
The normal situation is always reliability 255/255
01-16-2013 04:02 AM
Hi Jozef,
leolaohoo is right.
Most likely the issue is layer-1 related. Log message also points to the layer-1 failure:
%ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet101/1/7 is down (Link failure)
Since the server now works fine on the other port - please check the cable/sfp module on the FEX - if their replacement won't fix the issue please open the TAC case.
HTH,
Alex
04-12-2013 01:51 PM
Hi Jozef,
Were you able to find a resoution to this issue that was not related to the cable/sfp module? Was something on the switchport or switch misconfigured that caused the issue? We are seeing the same problem in our enviroment. Any help would be greatly appreciated.
-Jessy
04-12-2013 05:01 PM
Post the output of the relevant commands.
04-15-2013 07:58 AM
cisco answer I have been looking further into this and found the following bug, which has quite some similarities, although it is not an exact match: CSCuc51083 'Down' 2232 interfaces do not come 'Up' after ISSU to 5.2(1)n1(1b) => what is the same: - same NX-OS version with the problem - exact same type of FEX: N5K-C2232TM - is that also here only reload of the FEX solves this problem. - Trying to attach any other type of device to the affected port, doesn't help, not able to bring up any of these. What is different: is the trigger, in this bug an ISSU, while in your case the port was in use and then went down. In your case I saw last upgrade was: ----- reset reason for Supervisor-module 1 (from Supervisor in slot 1) --- 1) At 363117 usecs after Wed Oct 3 12:04:31 2012 Reason: Disruptive upgrade Service: Version: 5.2(1)N1(1) => knowing these similarities, it's quite possible that also some other sort of trigger exists other than an ISSU which could hit this defect, so I believe we indeed hit the same bug. Hence I would recommend on the long term to evaluate an upgrade to a fixed version of this bug. (short term fix is a reload of the FEX).
04-15-2013 04:05 PM
Jozef,
I will see if we have the same bug. Thank you for the quick response.
-Jessy
Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: