Cisco Support Community
cancel
Showing results for 
Search instead for 
Did you mean: 
Announcements

Welcome to Cisco Support Community. We would love to have your feedback.

For an introduction to the new site, click here. If you'd prefer to explore, try our test area to get started. And see here for current known issues.

New Member

All 5508 controllers just rebooted

Curious if anyone has seen this.

All of our 5508 Wireless Controllers have just rebooted across our UK sites, all running version 7.4.110.0, all rebooted at the same time for no obvious reason. Our old 4402 controller didnt reboot.

Everyone's tags (2)
25 REPLIES
Hall of Fame Super Gold

All 5508 controllers just rebooted

Did you find any crashinfo file?

New Member

Re: All 5508 controllers just rebooted

Sysinfo on all controllers shows:
Last Reset....................................... Software reset

I have pasted the first part of the crashinfo from 3 controllers below but seem identical.

All rebooted at same time (within a few seconds)

************************************************************
*             Start Cisco Crash Handler                *
************************************************************
Sys Name:  Cisco-HQ-5508-Grnd-C
Model:     AIR-CT5508-K9
Version:   7.4.110.0
Timestamp: Wed Jan  8 09:04:58 2014
SystemUpTime:  35 days 13 hrs 25 mins 39 secs
signal:    11
pid:       1070
TID:       1456812880
Task Name: emWeb
Reason:  System Crash
si_signo:  11
si_errno:  0
si_code:   1
si_addr:   0x0
timer tcb:   0x83d
timer cb:    0x10e76bd0 ('alarmSendMsgToMsgTask+48')
timer arg1:  0x0
timer arg2:  0x0

Long time taken timer call back inforamtion:
Time Stamp:  Wed Jan  8 09:04:58 2014
timer cb  :  10e76bd0p('alarmSendMsgToMsgTask+48')
Duration  : 320951 usecs, cbCount= 1

------------------------------------------------------------
Analysis of Failure:


  Software Failed on instruction at :
pc = 0x1206c7b0 (osapiStrlcpy+72), ra = 0x10af5e38 (osapiStrlcpy+72)


  Software Failed while accessing the data located at :0x0

------------------------------------------------------------


************************************************************
*             Start Cisco Crash Handler                *
************************************************************
Sys Name:  Cisco-GLA-5508-Glasgow
Model:     AIR-CT5508-K9
Version:   7.4.110.0
Timestamp: Wed Jan  8 09:04:49 2014
SystemUpTime:  36 days 23 hrs 33 mins 40 secs
signal:    11
pid:       1070
TID:       1435497296
Task Name: emWeb
Reason:  System Crash
si_signo:  11
si_errno:  0
si_code:   1
si_addr:   0x0
timer tcb:   0x83c
timer cb:    0x10e76bd0 ('alarmSendMsgToMsgTask+48')
timer arg1:  0x0
timer arg2:  0x0

Long time taken timer call back inforamtion:
Time Stamp:  Wed Jan  8 09:04:49 2014
timer cb  :  10e76bd0p('alarmSendMsgToMsgTask+48')
Duration  : 255209 usecs, cbCount= 2

------------------------------------------------------------
Analysis of Failure:


  Software Failed on instruction at :
pc = 0x1206c7b0 (osapiStrlcpy+72), ra = 0x10af5e38 (osapiStrlcpy+72)


  Software Failed while accessing the data located at :0x0

------------------------------------------------------------


************************************************************
*             Start Cisco Crash Handler                *
************************************************************
Sys Name:  Cisco-ARW-5508
Model:     AIR-CT5508-K9
Version:   7.4.110.0
Timestamp: Wed Jan  8 09:04:46 2014
SystemUpTime:  36 days 23 hrs 36 mins 42 secs
signal:    11
pid:       1070
TID:       1457722192
Task Name: emWeb
Reason:  System Crash
si_signo:  11
si_errno:  0
si_code:   1
si_addr:   0x0
timer tcb:   0x7fd
timer cb:    0x1102a950 ('nfaSyncMsgSendToTask+272')
timer arg1:  0x0
timer arg2:  0x0

Long time taken timer call back inforamtion:
Time Stamp:  Wed Jan  8 09:04:46 2014
timer cb  :  1102a950p('nfaSyncMsgSendToTask+272')
Duration  : 93038 usecs, cbCount= 1

------------------------------------------------------------
Analysis of Failure:


  Software Failed on instruction at :
pc = 0x1206c7b0 (osapiStrlcpy+72), ra = 0x10af5e38 (osapiStrlcpy+72)


  Software Failed while accessing the data located at :0x0

------------------------------------------------------------

VIP Purple

Re: All 5508 controllers just rebooted

Hi,

1. Remove the consle cable from Controller.

2.wait for few minutes then reboot the controller after reboot check again.

3. Update the Firmware on WLCs.

4. If still persists then raise a TAC case, it may be ahardware failure so RMA'd.

also chekc this bug: https://tools.cisco.com/bugsearch/bug/CSCuh55653

Regards

New Member

Hello all.Today I realized

Hello all.

Today I realized that yesterday I've got a reload at 11.00am in my primary WLC.

(wlc1) >show sysinfo

Manufacturer's Name.............................. Cisco Systems Inc.
Product Name..................................... Cisco Controller
Product Version.................................. 8.0.115.0
Bootloader Version............................... 1.0.20
Field Recovery Image Version..................... 7.6.101.1
Firmware Version................................. FPGA 1.7, Env 1.8, USB console 2.2
Build Type....................................... DATA + WPS

System Name...................................... wlc1
System Location..................................
System Contact...................................
System ObjectID.................................. 1.3.6.1.4.1.9.1.1069
Redundancy Mode.................................. Disabled
IP Address....................................... 10.2.0.68
IPv6 Address..................................... ::
Last Reset....................................... Software reset


I've searched for syslog messages but got no lucky

Nov 17 11:12:30 wlc1: *apfMsConnTask_4: Nov 17 11:12:30.820: #APF-3-CHECK_SUPP_RATES_FAILED: apf_utils.c:418 Could not check supported rates. Missing Supported Rate. Length :0. Mobile MAC: 50:cc:f8:72:3b:45.
Nov 17 11:12:31 wlc1: *apfMsConnTask_4: Nov 17 11:12:31.408: #APF-3-CHECK_SUPP_RATES_FAILED: apf_utils.c:418 Could not check supported rates. Missing Supported Rate. Length :0. Mobile MAC: c0:f2:fb:d6:89:d4.
Nov 17 11:16:13 wlc1: *nim_t: Nov 17 11:16:12.869: #SIM-3-PORT_UP: sim.c:13485 Physical port 2 is up!.
Nov 17 11:16:13 wlc1: *nim_t: Nov 17 11:16:12.869: #SIM-3-PORT_UP: sim.c:13485 Physical port 3 is up!.
Nov 17 11:16:13 wlc1: *nim_t: Nov 17 11:16:12.870: #SIM-3-PORT_UP: sim.c:13485 Physical port 4 is up!.

How can I check the issue for the "software reset"?
In the GUI I've checked for Management - Tech Support - Controller Crash,
but got no messages (***** End of Crash File dump *****)

any tip to figure out what happened?

VIP Purple

I am 99% sure this is a bug

I am 99% sure this is a bug in 8.0.115.0 code, only TAC can analyze (if you provided show tech from this WLC) and tell exact reason.

Since this is deffered code, i would upgrade to 8.0.121.0 straight away.

HTH

Rasika

New Member

hello. this weekend I've got

hello. this weekend I've got another crash on my primary WLC..

From GUI mgmt- tech support - controller crash, I've got nothing... :(

From CLI I've just got software reset.. please see above..

(wlc1) >show sysinfo     

Manufacturer's Name.............................. Cisco Systems Inc.
Product Name..................................... Cisco Controller
Product Version.................................. 8.0.121.0
Bootloader Version............................... 1.0.20
Field Recovery Image Version..................... 7.6.101.1
Firmware Version................................. FPGA 1.7, Env 1.8, USB console 2.2
Build Type....................................... DATA + WPS

System Name...................................... wlc1
System Location..................................
System Contact...................................
System ObjectID.................................. 1.3.6.1.4.1.9.1.1069
Redundancy Mode.................................. Disabled
IP Address....................................... 10.2.0.68
IPv6 Address..................................... ::
Last Reset....................................... Software reset

VIP Purple

Most probably this may due to

Most probably this may due to a software bug.

I would suggest you to move onto 8.0.132.0 (8.0MR3) or even go with upcoming 8.0MR4 when it releases.

https://supportforums.cisco.com/discussion/13014826/80mr4-beta-availability

If you look at fixes available in MR3 & MR4 you will be surprise how many defects are there in the current code you running.

HTH

Rasika 

TAC is now recommending 8.0

TAC is now recommending 8.0.133.0 for most WLCs. It even shows a golden star on Cisco Website due to stability. So you might want to go straight to 133. Seems like soon they will go to .135

Hall of Fame Super Gold

Product Version..............

Product Version.................................. 8.0.121.0

I wouldn't recommend anyone use this version of firmware.  It's not "stable" (I could use different adjective but this is the most appropriate).  8.0.133.0 or 8.1.131.0 are stable.  

New Member

We have never had this issue

We have never had this issue again once we upgraded the software on all controllers and now running 8.1.

Hall of Fame Super Gold

All 5508 controllers just rebooted

Hmmmm ... The crash info does not bode well.

On the upper right-hand corner of your screen you will see an option to create a TAC Case.  I suggest you do so.  In the meantime, get ready to upgrade to the latest 7.4 code currently available for download.

Hall of Fame Super Gold

Re: All 5508 controllers just rebooted

Task Name: emWeb

It's Cisco Bug ID CSCub88183.

And this is found in the Release Notes for both WiSM-2 and 5508 for firmware 7.4.110.0.

VIP Purple

All 5508 controllers just rebooted

Have you purchased all these controllers at the same time ? check with Cisco TAC if there any issue with the hardware depend on its manufacturing details. We are having 9x5508 in our environment & never came across a situation like this where all controller rebooted at once.(had to RMA specific 5508 batch due to silent crashes time to time)

Make sure your FUS 1.7.0.0 & go for 7.4MR2 (7.4.121.0) for the time being & see what TAC says on this. If I was you, I would push for RMA all controllers.

HTH

Rasika

**** Pls rate all useful responses ****

New Member

Re: All 5508 controllers just rebooted

Turns out it was Cisco Network Assistant when backing up

VIP Purple

Re: All 5508 controllers just rebooted

Agree with Leo....

Paste the crashinfo from WLC :
How..... Check this:

COntroller>Management>Tech Support> Controller Crash

Regards

Sent from Cisco Technical Support iPhone App

VIP Purple

Re: All 5508 controllers just rebooted

How many controllers all together get rebooted. Check all controllers "show sysinfo" to see last reload reason & time of the reboot match each other.

Post one of them "show sysinfo" output to see

HTH

Rasika

***** Pls rate all useful responses ****

Cisco Employee

All 5508 controllers just rebooted

Generally, RMA is not possible for software reset reasons. Only if there is dataplane crash that can't be fixed via software will be considered as hardware failure besides other true failures.

Last Reset....................................... Software reset

In this case, the crash task was due to emweb, it is possible NMS device may be trying to push an config to all WLCs caused this crash. Stack trace and coredump should speak about the actual reason.

VIP Purple

All 5508 controllers just rebooted

I am not saying you are hitting this bug. but in certain cases bugs are applicable to HW manufactured since certain date/time. So always worth to check.

CSCul68057

Symptom:

Wireless LAN Controller may encounter unexpected reload without crash file or coredump.

Console log output may include "reaperWatcher rebooting" and "!!!!! Watchdog detected LOCKUP !!!!!",

and there may be "#OSAPI-2-REAPER_WATCHER_INFO" message in syslog.

Conditions:

5508, 2504 or WiSM2 manufactured since 2012.

Known Affected Releases

7.4(110.0)

7.0(240.0)

7.5(102.0)

Known Fixed Releases

7.4.121.0

7.6.100.0

.

.

HTH

Rasika

Cisco Employee

All 5508 controllers just rebooted

Not trying to debate, at the same time trying to help out

This bug is about CF driver issue on WLC, it is not an hardware issue that require RMA, We can still upgrade the WLC to fixed code. And i don't see single RMA attached to this defect

Example of bug that require WLC RMA. It is not reoccuring so you don't find any external info.

https://tools.cisco.com/bugsearch/bug/CSCub00349

VIP Purple

Re: All 5508 controllers just rebooted

I agree with you.

What I wanted to say  is certain software bugs only applicable to particular harware versions like in this case. Did not try to say RMA is the solution

Rasika

Cisco Employee

Re: All 5508 controllers just rebooted

Sure. By the way most of the older AireOS releases will be deferred due to this bug within a week, field notice will be sent out.

Cisco Employee

All 5508 controllers just rebooted

post stack trace of the crash, need stack trace to get clue, sometimes that also does not help and require coredump of the crash. 7.4.121.0 or 7.4.111.X may have fix, if its an known bug.

New Member

Re: All 5508 controllers just rebooted

What I find strange is that all our 5508 controllers rebooted at the same time. I will schedule an upgrade on them all to the latest version

VIP Purple

All 5508 controllers just rebooted

Go for the 7.4.121.0 (7.4MR2). Also ensure that your FUS upgraded to 1.7.0.0 as well.

Rasika

New Member

All 5508 controllers just rebooted

Hi sprocket10,

I'm having the same issue. I have 2 WLC 5508 (HA) running v7.4.110.0.  Since September, never had any reboot problem until this week.

First, my primary WLC rebooted.  Today, both WLC rebooted around the same time.

Have you had the chance to upgrade and did that fix the problem ?

Thx

Tony

2145
Views
19
Helpful
25
Replies