How to tell if on board ethernet is failed on a 5508 wireless controller?

Unanswered Question
Oct 15th, 2014
User Badges:

We have a WLC 5508 that we use for mobility product development and customer demos.  When I insert a SFP (GLC-T) in any of the 8 slots the LED blinks amber a couple of times and then stays solid amber.  According to the manual this means "present with failure" though I cannot tell what exactly has failed. I have verified that the SFPs work when inserted into a 2960.  I have also tried different ethernet cables, rebooted multiple times (everything comes up OK on the bootup process) and have run every possible show command there is including show tech.  nothing gives any clue.  The port reports that the SFP is inserted (at records remove and insert in the logs) but that it is "line down".

Any tips on how I can find out what is actually failed on the controller?

Attachment: 
  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 0 (0 ratings)
Loading.
George Stefanick Wed, 10/15/2014 - 12:34
User Badges:
  • Purple, 4500 points or more
  • Community Spotlight Award,

    Best Publication, October 2015

Do you have another SFP to test with on the WLC?

jkirby Wed, 10/15/2014 - 12:42
User Badges:

Yes.  I've tried three GLC-T, all Cisco branded, and get the same result every time.  I can plug the same transceivers into any SFP slot on a 2960, attach my laptop, and they work.  The WLC even knows they are there but something is "failed" according to the amber LED.  Also tried rebooting with and without an SFP inserted but same result.

Leo Laohoo Thu, 10/16/2014 - 17:16
User Badges:
  • Super Gold, 25000 points or more
  • Hall of Fame,

    The Hall of Fame designation is a lifetime achievement award based on significant overall achievements in the community. 

  • Cisco Designated VIP,

    2017 LAN, Wireless

Console into the WLC and reboot the WLC.  

 

Post the entire bootup process.

jkirby Fri, 10/17/2014 - 07:27
User Badges:

Been there, done that.  There's nothing in the boot process that shows any failed anything.  But I"ll do it again and post here when I get a chance this morning.

<edit> 

Here's the boot sequence.  Hopefully you can see something in there that I'm missing:

WLCNG Boot Loader Version 1.0.1 (Built on Apr 11 2009 at 13:32:33 by cisco)
Board Revision 1.3 (SN: FCW1432L023, Type: AIR-CT5508-K9) (P)

Verifying boot loader integrity... OK.

OCTEON CN5645-NSP pass 2.1, Core clock: 600 MHz, DDR clock: 330 MHz (660 Mhz data rate)
FPGA Revision 1.3
Env FW Revision 1.6
USB Console Revision 1.27
DRAM:  1024 MB
Flash: 32 MB
Clearing DRAM........ done
Network: octeth0', octeth1
  ' - Active interface
  E - Environment MAC address override
CF Bus 0 (IDE): OK
IDE device 0:
 - Model: VM DFC 1GB Firm: 20090819 Ser#: VM1GB       00000181
 - Type: Hard Disk
 - Capacity: 967.6 MB = 0.9 GB (1981728 x 512)


Press <ESC> now to access the Boot Menu...

Loading primary image (7.6.120.0)
100%

36925545 bytes read
Launching...
init started: BusyBox v1.6.0 (2010-05-13 17:50:10 EDT) multi-call binary
starting pid 845, tty '': '/etc/init.d/rcS'
Set PLX switch MPS settings .............!!!!!!!
Detecting Hardware ...
set smp_affinity for irq 48
003f
DP from CGE5.0 ...
starting pid 1091, tty '/dev/ttyS0': '/usr/bin/gettyOrMwar'
Setting up ZVM
Exporting LD_LIBRARY_PATH

Cryptographic library self-test....passed!
XML config selected
Validating XML configuration
Read HA Config before validation
octeon_device_init: found 1 DPs
readCPUConfigData: cardid 0x6070001
Cisco is a trademark of Cisco Systems, Inc.
Software Copyright Cisco Systems, Inc. All rights reserved.

Cisco AireOS Version 7.6.120.0
Firmware Version FPGA 1.3, Env 1.6, USB console 1.27
Initializing OS Services: ok
Initializing Serial Services: ok
Initializing Network Services: ok
Initializing Licensing Services: ok

License daemon start initialization.....

License daemon running.....
Starting Statistics Service: ok
Starting ARP Services: ok
Starting Trap Manager: ok
Starting Network Interface Management Services: ok
Starting System Services: ok
Starting FIPS Features: ok : Not enabled
Starting Fastpath Hardware Acceleration: ok
Starting Fastpath Console redirect : ok
Starting Fastpath DP Heartbeat : ok
Fastpath CPU0.00: Starting Fastpath Application. SDK-1.8.0, build 269. Flags-[DUTY CYCLE] : ok
Fastpath CPU0.00: Initializing last packet received queue. Num of cores(10)
Fastpath CPU0.00: Init MBUF size: 1856, Subsequent MBUF size: 2040
Fastpath CPU0.00: Core 0 Initialization and FIPS self-test: ok
Fastpath CPU0.00: Initializing Timer...
Fastpath CPU0.00: Initializing Timer...done.
Fastpath CPU0.00: Initializing Timer...
Fastpath CPU0.00: Initializing NBAR AGING Timer...done.
Fastpath CPU0.01: Core 1 Initialization and FIPS self-test: ok
Fastpath CPU0.02: Core 2 Initialization and FIPS self-test: ok
Fastpath CPU0.03: Core 3 Initialization and FIPS self-test: ok
Fastpath CPU0.03: Received instruction to get link status
Fastpath CPU0.04: Core 4 Initialization and FIPS self-test: ok
Fastpath CPU0.05: Core 5 Initialization and FIPS self-test: ok
Fastpath CPU0.06: Core 6 Initialization and FIPS self-test: ok
Fastpath CPU0.07: Core 7 Initialization and FIPS self-test: ok
Fastpath CPU0.08: Core 8 Initialization and FIPS self-test: ok
Fastpath CPU0.09: Core 9 Initialization and FIPS self-test: ok
Starting Switching Services: ok
Starting QoS Services: ok
Starting Policy Manager: ok
Starting Data Transport Link Layer: ok
Starting Access Control List Services: ok
Starting System Interfaces: ok
Starting Client Troubleshooting Service: ok
Starting Management Frame Protection: ok
Starting Certificate Database: ok
Starting VPN Services: ok
Starting DNS Services: ok
Starting Licensing Services: ok
Starting Redundancy: ok
Starting LWAPP: ok
Starting CAPWAP: ok
Starting LOCP: ok                                                    +-----------------------------------+
Starting Security Services: ok                                       |                                   |
Starting Policy Manager: ok                                          |  Cannot open /dev/cu.KeySerial1!  |
Starting Authentication Engine: ok                                   |                                   |
Starting Mobility Management: ok                                     +-----------------------------------+
Starting Capwap Ping Component: ok
Starting AVC Services: ok
Starting Virtual AP Services: ok
Starting AireWave Director: ok
Starting Network Time Services: ok
Starting Cisco Discovery Protocol: ok
Starting Broadcast Services: ok
Starting Logging Services: ok
Starting DHCP Server: ok
Starting IDS Signature Manager: ok
Starting RFID Tag Tracking: ok
Starting RF Profiles: ok
Starting Power Supply and Fan Status Monitoring Service: ok
Starting Mesh Services:  ok
Starting TSM: ok
Starting CIDS Services: ok
Starting Ethernet-over-IP: ok
Starting DTLS server:  enabled in CAPWAP
Starting CleanAir: ok
Starting WIPS: ok
Starting SSHPM LSC PROV LIST: ok
Starting RRC Services: ok
Starting SXP Services: ok
Starting Alarm Services: ok
Starting FMC HS: ok
Starting IPv6 Services: ok
Starting Config Sync Manager : ok
Starting Hotspot Services: ok
Starting PMIP Services: ok
Starting Portal Server Services: ok
Starting mDNS Services: ok
Starting Management Services:
   Web Server:    CLI:    Secure Web: ok

(Cisco Controller)

Enter User Name (or 'Recover-Config' this one-time only to reset configuration to factory defaults)

User:  

 

Rasika Nayanajith Thu, 10/16/2014 - 23:41
User Badges:
  • Purple, 4500 points or more
  • Cisco Designated VIP,

    2017 Wireless

Try these commands on your WLC CLI & see if that gives any information

show port summary
show port detailed-info
debug fastpath cfgtool dump.sfp

HTH

Rasika

**** Pls rate all useful responses ****

jkirby Fri, 10/17/2014 - 07:25
User Badges:

This output is while the SFP is inserted and a known good cable plugged into a known good GigE port on a 2960 switch.

(Cisco Controller) >show port sum

           STP   Admin   Physical   Physical   Link   Link
Pr  Type   Stat   Mode     Mode      Status   Status  Trap     POE    SFPType    
-- ------- ---- ------- ---------- ---------- ------ ------- ------- ----------
1  Normal  Disa Enable  Auto       Auto       Down   Enable  N/A     1000BaseTX  
2  Normal  Disa Enable  Auto       Auto       Down   Enable  N/A     Not Present
3  Normal  Disa Enable  Auto       Auto       Down   Enable  N/A     Not Present
4  Normal  Disa Enable  Auto       Auto       Down   Enable  N/A     Not Present
5  Normal  Disa Enable  Auto       Auto       Down   Enable  N/A     Not Present
6  Normal  Disa Enable  Auto       Auto       Down   Enable  N/A     Not Present
7  Normal  Disa Enable  Auto       Auto       Down   Enable  N/A     Not Present
8  Normal  Disa Enable  Auto       Auto       Down   Enable  N/A     Not Present

There is no "show port detail" command.  I assume you meant "show interface detail" which produces this:

(Cisco Controller) >show interface detailed management

Interface Name................................... management
MAC Address...................................... d0:d0:fd:1f:a2:40
IP Address....................................... 10.100.1.14
IP Netmask....................................... 255.255.255.0
IP Gateway....................................... 10.100.1.1
External NAT IP State............................ Disabled
External NAT IP Address.......................... 0.0.0.0
VLAN............................................. untagged  
Quarantine-vlan.................................. 0
Active Physical Port............................. 1         
Primary Physical Port............................ 1         
Backup Physical Port............................. Unconfigured
DHCP Proxy Mode.................................. Global
Primary DHCP Server.............................. 10.100.1.1
Secondary DHCP Server............................ Unconfigured
DHCP Option 82................................... Disabled
IPv4 ACL......................................... Unconfigured
mDNS Profile Name................................ Unconfigured
AP Manager....................................... Yes
Guest Interface.................................. No
L2 Multicast..................................... Enabled

There is also no "debug fastpath cfgtool" so I did a "debug fastpath events" which produces this when the SFP is removed and reinserted:

FP0.04:(158104)SFP-INS_REM: REMOVE  SFP1 (phy17)
FP0.04:(158104)SFP INTERRUPT Port1 - SFP removed
FP0.04:(158104)Received command: FP_CMD_GET_PORT 1 (phy17) Returns: Admin=UP   Oper=DOWN Status=DOWN
FP0.05:(158112)SFP INTERRUPT Port1 - NEW SFP de-bounce
FP0.01:(158113)SFP-INS_REM: INSERT  SFP1 (phy17) 1000BaseTX
FP0.01:(158114)SFP INTERRUPT Port1 - NEW SFP detected
FP0.01:(158114)Received command: FP_CMD_GET_PORT 1 (phy17) Returns: Admin=UP   Oper=DOWN Status=DOWN

I'm not seeing anything that would indicate an obvious failure.

 

Rasika Nayanajith Fri, 10/17/2014 - 13:14
User Badges:
  • Purple, 4500 points or more
  • Cisco Designated VIP,

    2017 Wireless

Do you have any other SFP (1000BastTX) to test out see if that make any different. Also if you could check with a fibre SFP to see whether interface come up. If nothing works then I would RMA this WLC.

I would try one more thing before RMA, that is upgrade FUS of this WLC to 1.9.0.0 (via service port you should be able to do this) & upgrade AireOS image to 7.6.130.0.

http://www.cisco.com/c/en/us/td/docs/wireless/controller/release/notes/fus_rn_OL-31390-01.html

Below command should work as it is on certain model/sofware version of controllers.

debug fastpath cfgtool dump.sfp

Here is a proof from 7.4.x running on 5508 gives that info

http://mrncciew.com/2014/08/18/wlc-sfp-info/

Pls do not forget to rate our responses if that is useful to you.

HTH

Rasika

jkirby Fri, 10/17/2014 - 12:56
User Badges:

The problem has been resolved.  Indeed it was not one but two bad cables.  They worked with my laptop in GigE and FastE ports.  The worked between any two FastE ports.  They worked between a GigE and a FastE port.  But they would not work between any two GigE ports, go figure.  So I clipped the ends and told my engineer to re-terminate them. 

 

So anyway, thanks for all the advice.

jk

Actions

This Discussion

 

 

Trending Topics: Other Wireless Mobility

client could not be authenticated
Network Analysis Module (NAM) Products
Cisco 6500 nam
reason 440 driver failure
Cisco password cracker
Cisco Wireless mode