Solved: WLC randomly restarts! Please Help!

Craddockc · ‎02-14-2012

Hello,

I am currently using a 5508 Cisco WLC with software version 6.0.196.0. It has been running smoothly for the past 1.5 years, but have recently discovered that the WLC will randomly restarts itself. I know this because I have seen the disruptions in our wireless network, as well as seen the Uptime counter reset itself. I have tried looking at the trap logs to see if there is any significant network traffic that may be causing this, but from what I can tell there is nothing out of the oridnary. The restarts happen randomly at random times. I have never seen a Cisco device behave this way, not a switch, a router or security appliance for that fact. Can anyone help me to figure out what may be causing the WLC to just reboot itself? Any help you can provide would be greatly appreciated. Thank you.

Chris.

daviwatk · ‎02-15-2012

This crash is related to several different bugs in the 6.0.182.0 and 6.0.196.0 releases. "Most" of them are resolved in 6.0.202.0, however I would suggest (if your infrastructure allows) moving to the latest 7.0.230.0 release for this WLC. I think you will find that the WLC will stabalize.

To completely confirm "which" one, we would need to see the "entire" output of the crash file. Here are some examples of bugs it is possible you are hitting.

CSCtj21464

CSCte43508

CSCtg42711

Bottom line; there are "numerous" crashes that can be encountered on 5508s running those later 6.0 releases and I would suggest an upgrade. Upgrade to 7.0.230.0, clear all logs (traplogs, msglogs, and crashlogs) then monitor.

View solution in original post

daviwatk · ‎02-15-2012

I would highly suggest you upgrade your WLC immediately to remedy this issue. There isn't much else to look at. Again, these older 6.0 versions have numerous problems with crashes in our 5500 series and I would not even attempt to resolve/troubleshoot this issue with your existing version 6.0.196.0. "If" you are having crash problems after the update, then definitely open a case with TAC as something else is clearly wrong.

Again, I would update to 7.0.230.0, clear msglogs/traplogs/crashlogs and continue to monitor.

Release notes are here, but I have anchored this link directly to the list of compatibility of APs and this release. Unless you are using some very old models, like 1000 series APs (see matrix), then you should have no issues moving to this latest 7.0 release.

http://www.cisco.com/en/US/docs/wireless/controller/release/notes/crn7_0_230_0.html#wp784295

View solution in original post

George Stefanick · ‎02-14-2012

Hey Craddockc,

I see its your first post on CSC, welcome.

Did you check the controller crash log ? This would be the first place I would peek.

GO-->MANAGEMENT-->TECH SUPPORT--> CONTROLLER CRASH LOGS

Do you have anything in there, if so please post.

"Satisfaction does not come from knowing the solution, it comes from knowing why." - Rosalind Franklin
___________________________________________________________

Craddockc · ‎02-14-2012

George,

Thank you for your reply. Yes I have the contents of the crash log; however, it is very verbose and to me is almost unitelligible. I am not sure how to even start making sense of it. Would you like me to post the contents of the log in the next reply? Thanks.

Chris.

George Stefanick · ‎02-14-2012

Chris,

Yea I feel your pain. I might suggest posting it. We have a number of TAC engineers that troll the forums and they could help us decode it. If you have a support contract on your WLC you might want to open a case with TAC as well..

"Satisfaction does not come from knowing the solution, it comes from knowing why." - Rosalind Franklin
___________________________________________________________

Craddockc · ‎02-15-2012

Justin, George,

Thank you both for your replies and suggestions.

George,

I will go ahead and post the crash dump here soon.

Justin,

The temp sensor on the Controller reads 35C, which is about 95F, which isnt excessively hot. I havent touched it to see if it feels warmer then that though. Great suggestion is trying to reseat the RAM, perhaps some chip creep is occurring there. I will try reseating the RAM modules and see if that corrects the problem.

Craddockc · ‎02-15-2012

George, Justin,

Taking another look at the crash dump, I am noticing a pattern. I wont post the entire dump file as it is large and has some IP info in it, but will post what I feel is most relevent. For some reason this broffu_fp_dapi_cmd.c:2124 whatever is is, seems to be the problem.

Beginning of Crash File: mwar_dump2.crash
--------------------------------------------------------

Dumping registers
CPU Registers:
$ 0   : 00000000 8112bcb0 00000000 123523a0
$ 4   : 00000001 1af79cb0 169c6212 00000000
$ 8   : 0000004a 00000020 00000000 82f60000
$12   : 00000000 80000010 811533d8 00000000
$16   : 00000001 152389c0 00000001 00000001
$20   : 1234e470 11b60000 1233e470 151d0000
$24   : 00000000 114e9c00
$28   : 1232e470 721fe7e0 721fe7e0 1001468c
Hi    : 00000001
Lo    : 9999999a
Status: 00000000
Cause : 00000000
BadAddr : 00000000

Stack :
        00000000 00000000 00000000 00000000
        00000000 00000000 00000000 00000000
        00000000 00000000 00000000 00000000
        00000000 00000000 00000000 00000000
        00000000 00000000 00000000 00000000
        00000000 00000000 00000000 00000000
        00000000 00000000 00000000 00000000
        00000000 00000000 00000000 00000000
        00000000 00000000 00000000 00000000
        00000000 00000000 00000000 00000000
        00000000 12360000 00000000 11b60000
        00000000 00000001 00000000 00000001
        00000000 1234e470 00000000 11b60000
        00000000 1233e470 00000000 1232e470
        00000000 1002d6f8 00000000 114ee7c8
        00000000 1ad2bc0c 00000000 00000000
************************************************************
*             Start Cisco Crash Handler                *
************************************************************
Sys Name: Cisco_93:f1:84
Model:     AIR-CT5508-K9
Version:   6.0.196.0
Timestamp: Mon Feb 13 16:12:20 2012
SystemUpTime: 3 days 6 hrs 10 mins 26 secs
pid:       428
TID:       1802349
Task Name: osapiReaper
User Crash: The system has encountered a fatal condition at broffu_fp_dapi_cmd.c:2124

------------------------------------------------------------
Analysis of Failure:

Software was stopped for the following reason:
The system has encountered a fatal condition at broffu_fp_dapi_cmd.c:2124

Beginning of Crash File: mwar_dump3.crash
--------------------------------------------------------

Dumping registers
CPU Registers:
$ 0   : 00000000 8112bcb0 00000000 123523a0
$ 4   : 00000001 1af9c900 169c6212 00000000
$ 8   : 0000004a 00000020 00000000 82f60000
$12   : 00000000 80000010 811533d8 00000000
$16   : 00000001 152389c0 00000001 00000001
$20   : 1234e470 11b60000 1233e470 151d0000
$24   : 00000000 114e9c00
$28   : 1232e470 721fe7e0 721fe7e0 1001468c
Hi    : 00000001
Lo    : 9999999a
Status: 00000000
Cause : 00000000
BadAddr : 00000000

Stack :
        00000000 00000000 00000000 1232e470
        00000000 12360000 721fe8b8 00010000
        2ae9f0a4 00000001 00000000 00020000
        00000000 114ce258 00000000 00000000
        00000000 12360000 00000000 00ff3280
        ffffffff ffffffff 00000000 1233e470
        00000000 00000001 00000000 12360000
        00000000 12360000 00000000 00010000
        00000000 1232e470 00000000 12360000
        00000000 1002d4c8 00000000 1002d480
        57445744 00000675 00000000 11b60000
        00000000 00000002 00000000 12365de2
        00000000 0000000a 00000000 11540000
        00000000 11b60000 00000000 1232e470
        00000000 120e0000 00000000 1002d398
        00000000 1232e470 00000000 00020000
************************************************************
*             Start Cisco Crash Handler                *
************************************************************
Sys Name: Cisco_93:f1:84
Model:     AIR-CT5508-K9
Version:   6.0.196.0
Timestamp: Fri Feb 10 10:00:58 2012
SystemUpTime: 2 days 19 hrs 20 mins 30 secs
pid:       428
TID:       1802349
Task Name: osapiReaper
User Crash: The system has encountered a fatal condition at broffu_fp_dapi_cmd.c:2124

------------------------------------------------------------
Analysis of Failure:

Software was stopped for the following reason:
The system has encountered a fatal condition at broffu_fp_dapi_cmd.c:2124

Beginning of Crash File: mwar_dump4.crash
--------------------------------------------------------

Dumping registers
CPU Registers:
$ 0   : 00000000 8112bcb0 00000000 123523a0
$ 4   : 00000001 1af84360 169c6212 00000000
$ 8   : 0000004a 00000020 00000000 82f60000
$12   : 00000000 80000010 811533d8 00000000
$16   : 00000001 152389c0 00000001 00000001
$20   : 1234e470 11b60000 1233e470 151d0000
$24   : 00000000 114e9c00
$28   : 1232e470 721fe7e0 721fe7e0 1001468c
Hi    : 00000001
Lo    : 9999999a
Status: 00000000
Cause : 00000000
BadAddr : 00000000

Stack :
        00000000 00000000 00000000 00000000
        00000000 00000000 00000000 00000000
        00000000 00000000 00000000 00000000
        00000000 00000000 00000000 00000000
        00000000 00000000 00000000 00000000
        00000000 00000000 00000000 00000000
        00000000 00000000 00000000 00000000
        00000000 00000000 00000000 00000000
        00000000 00000000 00000000 00000000
        00000000 00000000 00000000 00000000
        00000000 12360000 00000000 11b60000
        00000000 00000001 00000000 00000001
        00000000 1234e470 00000000 11b60000
        00000000 1233e470 00000000 1232e470
        00000000 1002d6f8 00000000 114ee7c8
        00000000 1ad2bc0c 00000000 00000000
************************************************************
*             Start Cisco Crash Handler                *
************************************************************
Sys Name: Cisco_93:f1:84
Model:     AIR-CT5508-K9
Version:   6.0.196.0
Timestamp: Tue Feb 7 14:36:18 2012
SystemUpTime: 17 days 23 hrs 18 mins 20 secs
pid:       428
TID:       1802349
Task Name: osapiReaper
User Crash: The system has encountered a fatal condition at broffu_fp_dapi_cmd.c:2124

------------------------------------------------------------
Analysis of Failure:

Software was stopped for the following reason:
The system has encountered a fatal condition at broffu_fp_dapi_cmd.c:2124

daviwatk · ‎02-15-2012

This crash is related to several different bugs in the 6.0.182.0 and 6.0.196.0 releases. "Most" of them are resolved in 6.0.202.0, however I would suggest (if your infrastructure allows) moving to the latest 7.0.230.0 release for this WLC. I think you will find that the WLC will stabalize.

To completely confirm "which" one, we would need to see the "entire" output of the crash file. Here are some examples of bugs it is possible you are hitting.

CSCtj21464

CSCte43508

CSCtg42711

Bottom line; there are "numerous" crashes that can be encountered on 5508s running those later 6.0 releases and I would suggest an upgrade. Upgrade to 7.0.230.0, clear all logs (traplogs, msglogs, and crashlogs) then monitor.

Justin Kurynny · ‎02-15-2012

Have you recently moved the controller or does it feel excessively warm to the touch? Reseating RAM has fixed this problem for me in the past. There are two modules in the 5508.

Sent from Cisco Technical Support iPhone App

Craddockc · ‎02-15-2012

David,

Thank you for your reply. After some further research on the support forums it appears someone else had the same problem. It appears an upgrade as you suggested fixed the issue. The thread is posted here https://supportforums.cisco.com/thread/2112228

Should I go head with downloading the latest IOS image and upgrade? Or should I post the crash dump here anyway? Or should I open a TAC request before I do anything? I am kind of treading in unkown waters here at the moment, as up to this point I have never had to take an issue to these depths. I appreciate any information you can provide. Thanks.

Chris.

daviwatk · ‎02-15-2012

I would highly suggest you upgrade your WLC immediately to remedy this issue. There isn't much else to look at. Again, these older 6.0 versions have numerous problems with crashes in our 5500 series and I would not even attempt to resolve/troubleshoot this issue with your existing version 6.0.196.0. "If" you are having crash problems after the update, then definitely open a case with TAC as something else is clearly wrong.

Again, I would update to 7.0.230.0, clear msglogs/traplogs/crashlogs and continue to monitor.

Release notes are here, but I have anchored this link directly to the list of compatibility of APs and this release. Unless you are using some very old models, like 1000 series APs (see matrix), then you should have no issues moving to this latest 7.0 release.

http://www.cisco.com/en/US/docs/wireless/controller/release/notes/crn7_0_230_0.html#wp784295

Craddockc · ‎02-15-2012

David, Justin, George,

Thank you very much for assisting me in this issue. Coming to this support forum has proved to be an invaluable experience. I very much appreciate you guys taking the time to help me and to impart the knowledge you have about this issue. I look forward to coming back here!

I will go ahead and upgrade my WLC and clear the crash dump and logs and continue to monitor. Thanks again very much for all your help!

Chris.

Craddockc · ‎05-16-2012

Just wanted to let you guys know that I upgraded my WLC SW version to 7.2.103.0 exactly 30 days ago, NO ISSUES as of yet! Thanks again for all your help!

Chris.