All 5508 controllers just rebooted

sprocket10 · ‎01-08-2014

Curious if anyone has seen this.

All of our 5508 Wireless Controllers have just rebooted across our UK sites, all running version 7.4.110.0, all rebooted at the same time for no obvious reason. Our old 4402 controller didnt reboot.

Leo Laohoo · ‎01-08-2014

Did you find any crashinfo file?

sprocket10 · ‎01-09-2014

Sysinfo on all controllers shows:
Last Reset....................................... Software reset

I have pasted the first part of the crashinfo from 3 controllers below but seem identical.

All rebooted at same time (within a few seconds)

************************************************************
*             Start Cisco Crash Handler                *
************************************************************
Sys Name: Cisco-HQ-5508-Grnd-C
Model:     AIR-CT5508-K9
Version:   7.4.110.0
Timestamp: Wed Jan 8 09:04:58 2014
SystemUpTime: 35 days 13 hrs 25 mins 39 secs
signal:    11
pid:       1070
TID:       1456812880
Task Name: emWeb
Reason: System Crash
si_signo: 11
si_errno: 0
si_code:   1
si_addr:   0x0
timer tcb:   0x83d
timer cb:    0x10e76bd0 ('alarmSendMsgToMsgTask+48')
timer arg1: 0x0
timer arg2: 0x0

Long time taken timer call back inforamtion:
Time Stamp: Wed Jan 8 09:04:58 2014
timer cb : 10e76bd0p('alarmSendMsgToMsgTask+48')
Duration : 320951 usecs, cbCount= 1

------------------------------------------------------------
Analysis of Failure:

Software Failed on instruction at :
pc = 0x1206c7b0 (osapiStrlcpy+72), ra = 0x10af5e38 (osapiStrlcpy+72)

Software Failed while accessing the data located at :0x0

------------------------------------------------------------

************************************************************
*             Start Cisco Crash Handler                *
************************************************************
Sys Name: Cisco-GLA-5508-Glasgow
Model:     AIR-CT5508-K9
Version:   7.4.110.0
Timestamp: Wed Jan 8 09:04:49 2014
SystemUpTime: 36 days 23 hrs 33 mins 40 secs
signal:    11
pid:       1070
TID:       1435497296
Task Name: emWeb
Reason: System Crash
si_signo: 11
si_errno: 0
si_code:   1
si_addr:   0x0
timer tcb:   0x83c
timer cb:    0x10e76bd0 ('alarmSendMsgToMsgTask+48')
timer arg1: 0x0
timer arg2: 0x0

Long time taken timer call back inforamtion:
Time Stamp: Wed Jan 8 09:04:49 2014
timer cb : 10e76bd0p('alarmSendMsgToMsgTask+48')
Duration : 255209 usecs, cbCount= 2

------------------------------------------------------------
Analysis of Failure:

Software Failed on instruction at :
pc = 0x1206c7b0 (osapiStrlcpy+72), ra = 0x10af5e38 (osapiStrlcpy+72)

Software Failed while accessing the data located at :0x0

------------------------------------------------------------

************************************************************
*             Start Cisco Crash Handler                *
************************************************************
Sys Name: Cisco-ARW-5508
Model:     AIR-CT5508-K9
Version:   7.4.110.0
Timestamp: Wed Jan 8 09:04:46 2014
SystemUpTime: 36 days 23 hrs 36 mins 42 secs
signal:    11
pid:       1070
TID:       1457722192
Task Name: emWeb
Reason: System Crash
si_signo: 11
si_errno: 0
si_code:   1
si_addr:   0x0
timer tcb:   0x7fd
timer cb:    0x1102a950 ('nfaSyncMsgSendToTask+272')
timer arg1: 0x0
timer arg2: 0x0

Long time taken timer call back inforamtion:
Time Stamp: Wed Jan 8 09:04:46 2014
timer cb : 1102a950p('nfaSyncMsgSendToTask+272')
Duration : 93038 usecs, cbCount= 1

------------------------------------------------------------
Analysis of Failure:

Software Failed on instruction at :
pc = 0x1206c7b0 (osapiStrlcpy+72), ra = 0x10af5e38 (osapiStrlcpy+72)

Software Failed while accessing the data located at :0x0

------------------------------------------------------------

Sandeep Choudhary · ‎01-09-2014

Hi,

1. Remove the consle cable from Controller.

2.wait for few minutes then reboot the controller after reboot check again.

3. Update the Firmware on WLCs.

4. If still persists then raise a TAC case, it may be ahardware failure so RMA'd.

also chekc this bug: https://tools.cisco.com/bugsearch/bug/CSCuh55653

Regards

Bruno Dinis · ‎11-18-2015

Hello all.

Today I realized that yesterday I've got a reload at 11.00am in my primary WLC.

(wlc1) >show sysinfo

Manufacturer's Name.............................. Cisco Systems Inc.
Product Name..................................... Cisco Controller
Product Version.................................. 8.0.115.0
Bootloader Version............................... 1.0.20
Field Recovery Image Version..................... 7.6.101.1
Firmware Version................................. FPGA 1.7, Env 1.8, USB console 2.2
Build Type....................................... DATA + WPS

System Name...................................... wlc1
System Location..................................
System Contact...................................
System ObjectID.................................. 1.3.6.1.4.1.9.1.1069
Redundancy Mode.................................. Disabled
IP Address....................................... 10.2.0.68
IPv6 Address..................................... ::
Last Reset....................................... Software reset

I've searched for syslog messages but got no lucky

Nov 17 11:12:30 wlc1: *apfMsConnTask_4: Nov 17 11:12:30.820: #APF-3-CHECK_SUPP_RATES_FAILED: apf_utils.c:418 Could not check supported rates. Missing Supported Rate. Length :0. Mobile MAC: 50:cc:f8:72:3b:45.
Nov 17 11:12:31 wlc1: *apfMsConnTask_4: Nov 17 11:12:31.408: #APF-3-CHECK_SUPP_RATES_FAILED: apf_utils.c:418 Could not check supported rates. Missing Supported Rate. Length :0. Mobile MAC: c0:f2:fb:d6:89:d4.
Nov 17 11:16:13 wlc1: *nim_t: Nov 17 11:16:12.869: #SIM-3-PORT_UP: sim.c:13485 Physical port 2 is up!.
Nov 17 11:16:13 wlc1: *nim_t: Nov 17 11:16:12.869: #SIM-3-PORT_UP: sim.c:13485 Physical port 3 is up!.
Nov 17 11:16:13 wlc1: *nim_t: Nov 17 11:16:12.870: #SIM-3-PORT_UP: sim.c:13485 Physical port 4 is up!.

How can I check the issue for the "software reset"?
In the GUI I've checked for Management - Tech Support - Controller Crash,
but got no messages (***** End of Crash File dump *****)

any tip to figure out what happened?

Rasika Nayanajith · ‎11-18-2015

I am 99% sure this is a bug in 8.0.115.0 code, only TAC can analyze (if you provided show tech from this WLC) and tell exact reason.

Since this is deffered code, i would upgrade to 8.0.121.0 straight away.

HTH

Rasika

Bruno Dinis · ‎05-23-2016

hello. this weekend I've got another crash on my primary WLC..

From GUI mgmt- tech support - controller crash, I've got nothing... :(

From CLI I've just got software reset.. please see above..

(wlc1) >show sysinfo

Manufacturer's Name.............................. Cisco Systems Inc.
Product Name..................................... Cisco Controller
Product Version.................................. 8.0.121.0
Bootloader Version............................... 1.0.20
Field Recovery Image Version..................... 7.6.101.1
Firmware Version................................. FPGA 1.7, Env 1.8, USB console 2.2
Build Type....................................... DATA + WPS

System Name...................................... wlc1
System Location..................................
System Contact...................................
System ObjectID.................................. 1.3.6.1.4.1.9.1.1069
Redundancy Mode.................................. Disabled
IP Address....................................... 10.2.0.68
IPv6 Address..................................... ::
Last Reset....................................... Software reset

Rasika Nayanajith · ‎05-23-2016

Most probably this may due to a software bug.

I would suggest you to move onto 8.0.132.0 (8.0MR3) or even go with upcoming 8.0MR4 when it releases.

https://supportforums.cisco.com/discussion/13014826/80mr4-beta-availability

If you look at fixes available in MR3 & MR4 you will be surprise how many defects are there in the current code you running.

HTH

Rasika

Eduardo Andres Barreto Rosales · ‎06-09-2016

TAC is now recommending 8.0.133.0 for most WLCs. It even shows a golden star on Cisco Website due to stability. So you might want to go straight to 133. Seems like soon they will go to .135

Leo Laohoo · ‎05-23-2016

Product Version.................................. 8.0.121.0

I wouldn't recommend anyone use this version of firmware. It's not "stable" (I could use different adjective but this is the most appropriate). 8.0.133.0 or 8.1.131.0 are stable.

sprocket10 · ‎11-19-2015

We have never had this issue again once we upgraded the software on all controllers and now running 8.1.

Leo Laohoo · ‎01-09-2014

Hmmmm ... The crash info does not bode well.

On the upper right-hand corner of your screen you will see an option to create a TAC Case. I suggest you do so. In the meantime, get ready to upgrade to the latest 7.4 code currently available for download.

Leo Laohoo · ‎01-09-2014

Task Name: emWeb

It's Cisco Bug ID CSCub88183.

And this is found in the Release Notes for both WiSM-2 and 5508 for firmware 7.4.110.0.

Rasika Nayanajith · ‎01-09-2014

Have you purchased all these controllers at the same time ? check with Cisco TAC if there any issue with the hardware depend on its manufacturing details. We are having 9x5508 in our environment & never came across a situation like this where all controller rebooted at once.(had to RMA specific 5508 batch due to silent crashes time to time)

Make sure your FUS 1.7.0.0 & go for 7.4MR2 (7.4.121.0) for the time being & see what TAC says on this. If I was you, I would push for RMA all controllers.

HTH

Rasika

**** Pls rate all useful responses ****

sprocket10 · ‎03-06-2014

Turns out it was Cisco Network Assistant when backing up