Cisco Support Community
cancel
Showing results for 
Search instead for 
Did you mean: 
New Member

WLC 5508 v6.0.202.0 reboot randomly.

Dear all,

I have two WLCs 5508 on version 6.0.202.0 and it reboot randomly in the last time, I have the following crash log information and I cannot interpret the information, so somebody has had experience with this crash

--------------------------------------------------------

Beginning of Crash File: mwar_dump2.crash

--------------------------------------------------------

Dumping registers

CPU Registers:

$ 0   : 00000000 8112be70 2ac30339 00004004

$ 4   : 0600ca00 00000000 00000004 2ac2c341

$ 8   : 00000000 7a3fefc1 00020000 7a3fefec

$12   : 00000000 50617369 81125650 00000000

$16   : 2ac2c33d 2ac30341 00000000 0000c00c

$20   : 00000040 00000004 7a3fefe8 12333c18

$24   : 81490000 114fad10                 

$28   : 12323c18 7a3fee80 00000007 108a9748

Hi    : 0000001f

Lo    : 33333362

Status: 00000000

Cause : 00000000

BadAddr : 00000000

Stack :

        0600ca00 00000000 00000000 00000000

        165f417c 00000000 00000000 80005002

        00000000 00000027 00000000 2ac2c321

        00000000 00000028 00000000 0000005e

        00000000 12323c18 00000000 165f417c

        00000000 00000000 00000000 00000001

        00000000 12323c18 00000000 7a3fefe8

        00000000 108ad670 00000000 108ad4cc

        00000000 00000001 00000000 12323c18

        7a3fefc0 7a3fefd0 7a3fefc1 7a3fefec

        00000000 1cd441fc 00000000 1a133d80

        00000000 165f4140 00000000 1a133d80

        00000000 165f4140 00000000 165f417c

        00000000 00000000 00000000 12323c18

        00000000 12323c18 00000000 0a31600a

        00000000 10273fe0 00000000 10274174

************************************************************

*             Start Cisco Crash Handler                *

************************************************************

Sys Name:           WLC

Model:              AIR-CT5508-K9

Version:            6.0.202.0

Timestamp:          Sat Aug  3 20:35:58 2013

SystemUpTime:           2 days 6 hrs 2 mins 37 secs

signal:             11

pid:                346

TID:                704556

Task Name:          spamApTask

Reason:           System Crash

si_signo:           11

si_errno:           0

si_code:            1

si_addr:            0xc

timer tcb:            0x221

timer cb:             0x1030e450 ('rrmTimerInit+352')

timer arg1:           0x1d07c728

timer arg2:           0x0

Long time taken timer call back inforamtion:

Time Stamp:           Thu Aug  1 23:01:39 2013

timer cb  :           0x10296400 ('fmcSendNmspOneSecResponse+704')

Duration  : 1000000 usecs, cbCount= 2

------------------------------------------------------------

Analysis of Failure:

  Software Failed on instruction at :

pc = 0x108a95c4 (acDecodeChangeStateEventRequest+1648), ra = 0x108a9748 (acDecodeChangeStateEventRequest+1648)

------------------------------------------------------------

System Stack

    Frame 0: 0x10012fd8: create_crash_dump+7244

    Frame 1: 0x10011cc8: create_crash_dump+2364

    Frame 2: 0x1000768c: sigsegv_handler+5960

    Frame 3: 0x114b15c0: SHA1Final+1494592

    Frame 4: 0x7a3feb70: SHA1Final+1762370544

    Frame 5: 0x108a95c4: acDecodeChangeStateEventRequest+1260

    Frame 6: 0x108ad670: acDecodeDiscoveryRequest+672

    Frame 7: 0x10273fe0: acCapwapSmInit+31256

    Frame 8: 0x1026b3c4: acPostDecodeConfigRequest+4988

    Frame 9: 0x10276c7c: capwapAcStatemachine+3004

    Frame 10: 0x108a4d8c: spamApReceiveTask+692

    Frame 11: 0x10640b70: debug_heartbeat+392

    Frame 12: 0x114a9f10: SHA1Final+1464208

    Frame 13: 0x1150a494: SHA1Final+1858836

------------------------------------------------------------

Semaphore and Mutex Usage

(Caller IP(instruction pointer of caller) Gives one more level

of depth in stack to track the Semaphore and Mutex operation)

   Mutex 'pgrp->groupingSem' @ 0x1d02b9d4  Created fp_main_task(rrmgroup.c:917)

      Locked by grouping Task(rrmgroup.c:978), Caller IP = 0x10640b70 (debug_heartbeat+392)

      1 tasks waiting for resource.

Stackdump from grouping Task

    Frame 0: 0x10007af4: sighup_handler+116

    Frame 1: 0x114b15c0: SHA1Final+1494592

    Frame 2: 0x755fef80: SHA1Final+1680582656

    Frame 3: 0x114ad1bc: SHA1Final+1477180

    Frame 4: 0x114abd9c: SHA1Final+1472028

    Frame 5: 0x114afd28: SHA1Final+1488296

    Frame 6: 0x114ab744: SHA1Final+1470404

    Frame 7: 0x10634b20: osapiMutexTake+1008

    Frame 8: 0x102f1cf0: groupCalcEntry+160

    Frame 9: 0x10640b70: debug_heartbeat+392

    Frame 10: 0x114a9f10: SHA1Final+1464208

    Frame 11: 0x1150a494: SHA1Final+1858836

          grouping Task, rrmgroup.c:978, Caller IP = 0x10640b70 (debug_heartbeat+392)

Stackdump from grouping Task

    Frame 0: 0x10007af4: sighup_handler+116

    Frame 1: 0x114b15c0: SHA1Final+1494592

    Frame 2: 0x755fef80: SHA1Final+1680582656

    Frame 3: 0x114ad1bc: SHA1Final+1477180

    Frame 4: 0x114abd9c: SHA1Final+1472028

    Frame 5: 0x114afd28: SHA1Final+1488296

    Frame 6: 0x114ab744: SHA1Final+1470404

    Frame 7: 0x10634b20: osapiMutexTake+1008

    Frame 8: 0x102f1cf0: groupCalcEntry+160

    Frame 9: 0x10640b70: debug_heartbeat+392

    Frame 10: 0x114a9f10: SHA1Final+1464208

    Frame 11: 0x1150a494: SHA1Final+1858836

   Mutex 'pgrp->groupingSem' @ 0x1d02be1c  Created fp_main_task(rrmgroup.c:917)

      Locked by grouping Task(rrmgroup.c:978), Caller IP = 0x10640b70 (debug_heartbeat+392)

      1 tasks waiting for resource.

Stackdump from grouping Task

    Frame 0: 0x10007af4: sighup_handler+116

    Frame 1: 0x114b15c0: SHA1Final+1494592

    Frame 2: 0x75ffef80: SHA1Final+1691068416

    Frame 3: 0x114ad1bc: SHA1Final+1477180

    Frame 4: 0x114abd9c: SHA1Final+1472028

    Frame 5: 0x114afd28: SHA1Final+1488296

    Frame 6: 0x114ab744: SHA1Final+1470404

    Frame 7: 0x10634b20: osapiMutexTake+1008

    Frame 8: 0x102f1cf0: groupCalcEntry+160

    Frame 9: 0x10640b70: debug_heartbeat+392

    Frame 10: 0x114a9f10: SHA1Final+1464208

    Frame 11: 0x1150a494: SHA1Final+1858836

          grouping Task, rrmgroup.c:978, Caller IP = 0x10640b70 (debug_heartbeat+392)

Stackdump from grouping Task

    Frame 0: 0x10007af4: sighup_handler+116

    Frame 1: 0x114b15c0: SHA1Final+1494592

    Frame 2: 0x75ffef80: SHA1Final+1691068416

    Frame 3: 0x114ad1bc: SHA1Final+1477180

    Frame 4: 0x114abd9c: SHA1Final+1472028

    Frame 5: 0x114afd28: SHA1Final+1488296

    Frame 6: 0x114ab744: SHA1Final+1470404

    Frame 7: 0x10634b20: osapiMutexTake+1008

    Frame 8: 0x102f1cf0: groupCalcEntry+160

    Frame 9: 0x10640b70: debug_heartbeat+392

    Frame 10: 0x114a9f10: SHA1Final+1464208

    Frame 11: 0x1150a494: SHA1Final+1858836

   Mutex 'rcb->semId' @ 0x1cd3679c  Created spamReceiveTask(spam_list.c:336)

      Locked by spamApTask(spam_api.c:500), Caller IP = 0x1026b0f4 (acPostDecodeConfigRequest+4268)

      2 tasks waiting for resource.

Stackdump from spamApTask

    Frame 0: 0x10007af4: sighup_handler+116

    Frame 1: 0x7a3fe330: SHA1Final+1762368432

    Frame 2: 0x114db504: SHA1Final+1666436

          dot11b, spam_api.c:500, Caller IP = 0x101aa3e0 (spamGetNextLrad+296)

          dot11a, spam_api.c:500, Caller IP = 0x101aa3e0 (spamGetNextLrad+296)

Stackdump from dot11b

    Frame 0: 0x10007af4: sighup_handler+116

    Frame 1: 0x114b15c0: SHA1Final+1494592

    Frame 2: 0x763feb60: SHA1Final+1695261664

    Frame 3: 0x114ad1bc: SHA1Final+1477180

    Frame 4: 0x114abd9c: SHA1Final+1472028

    Frame 5: 0x114afd28: SHA1Final+1488296

    Frame 6: 0x114ab690: SHA1Final+1470224

    Frame 7: 0x10634b20: osapiMutexTake+1008

    Frame 8: 0x101962bc: spamGetLCBFromMacTemp+396

    Frame 9: 0x101aa3e0: spamGetNextLrad+296

    Frame 10: 0x102e49a4: rrmClientBuildLradList+188

    Frame 11: 0x102e9c5c: rrmClientProcessPacket+556

    Frame 12: 0x102ddce8: rrmClientStateMachine+864

    Frame 13: 0x1030cec0: rrmClientTaskEntry+96

    Frame 14: 0x10640b70: debug_heartbeat+392

    Frame 15: 0x114a9f10: SHA1Final+1464208

    Frame 16: 0x1150a494: SHA1Final+1858836

Stackdump from dot11a

    Frame 0: 0x10007af4: sighup_handler+116

    Frame 1: 0x114b15c0: SHA1Final+1494592

    Frame 2: 0x759feb60: SHA1Final+1684775904

    Frame 3: 0x114ad1bc: SHA1Final+1477180

    Frame 4: 0x114abd9c: SHA1Final+1472028

    Frame 5: 0x114afd28: SHA1Final+1488296

    Frame 6: 0x114ab690: SHA1Final+1470224

    Frame 7: 0x10634b20: osapiMutexTake+1008

    Frame 8: 0x101962bc: spamGetLCBFromMacTemp+396

    Frame 9: 0x101aa3e0: spamGetNextLrad+296

    Frame 10: 0x102e49a4: rrmClientBuildLradList+188

    Frame 11: 0x102e9c5c: rrmClientProcessPacket+556

    Frame 12: 0x102ddce8: rrmClientStateMachine+864

    Frame 13: 0x1030cec0: rrmClientTaskEntry+96

    Frame 14: 0x10640b70: debug_heartbeat+392

    Frame 15: 0x114a9f10: SHA1Final+1464208

    Frame 16: 0x1150a494: SHA1Final+1858836

   Mutex 'sshpmTftpRuleSemaphorePtr' @ 0x1cc911b4  Created fp_main_task(sshpm.c:509)

      Locked by sshpmReceiveTask(sshpmrecv.c:1566), Caller IP = 0x10640b70 (debug_heartbeat+392)

      0 tasks waiting for resource.

Stackdump from sshpmReceiveTask

    Frame 0: 0x10007af4: sighup_handler+116

    Frame 1: 0x114b15c0: SHA1Final+1494592

    Frame 2: 0x7b7ff4b0: SHA1Final+1783344432

    Frame 3: 0x114ad1bc: SHA1Final+1477180

    Frame 4: 0x114abd9c: SHA1Final+1472028

    Frame 5: 0x114a8aac: SHA1Final+1458988

    Frame 6: 0x1129f790: d_async_queue_pop+132

    Frame 7: 0x1062d888: osapiMessageReceive+472

    Frame 8: 0x103a209c: sshpmReceiveTaskEntry+932

    Frame 9: 0x10640b70: debug_heartbeat+392

    Frame 10: 0x114a9f10: SHA1Final+1464208

    Frame 11: 0x1150a494: SHA1Final+1858836

   Mutex 'L7_transferSem' @ 0x1cb5d1bc  Created TransferTask(updcode.c:3781)

      Locked by TransferTask(updcode.c:3787), Caller IP = 0x10640b70 (debug_heartbeat+392)

      1 tasks waiting for resource.

Stackdump from TransferTask

    Frame 0: 0x10007af4: sighup_handler+116

    Frame 1: 0x114b15c0: SHA1Final+1494592

    Frame 2: 0x7e7ff380: SHA1Final+1833675776

    Frame 3: 0x114ad1bc: SHA1Final+1477180

    Frame 4: 0x114abd9c: SHA1Final+1472028

    Frame 5: 0x114afd28: SHA1Final+1488296

    Frame 6: 0x114ab744: SHA1Final+1470404

    Frame 7: 0x10634b20: osapiMutexTake+1008

    Frame 8: 0x10024430: L7_transfer_task+300

    Frame 9: 0x10640b70: debug_heartbeat+392

    Frame 10: 0x114a9f10: SHA1Final+1464208

    Frame 11: 0x1150a494: SHA1Final+1858836

          TransferTask, updcode.c:3787, Caller IP = 0x10640b70 (debug_heartbeat+392)

Stackdump from TransferTask

    Frame 0: 0x10007af4: sighup_handler+116

    Frame 1: 0x114b15c0: SHA1Final+1494592

    Frame 2: 0x7e7ff380: SHA1Final+1833675776

    Frame 3: 0x114ad1bc: SHA1Final+1477180

    Frame 4: 0x114abd9c: SHA1Final+1472028

    Frame 5: 0x114afd28: SHA1Final+1488296

    Frame 6: 0x114ab744: SHA1Final+1470404

    Frame 7: 0x10634b20: osapiMutexTake+1008

    Frame 8: 0x10024430: L7_transfer_task+300

    Frame 9: 0x10640b70: debug_heartbeat+392

    Frame 10: 0x114a9f10: SHA1Final+1464208

    Frame 11: 0x1150a494: SHA1Final+1858836

------------------------------------------------------------

   Memory Information

System Memory Information:

             Total System Memory: 929660928 bytes (886.65 Mb)

              Used System Memory: 341839872 bytes (326.02 Mb)

              Free System Memory: 587821056 bytes (560.63 Mb)

Heap information:

         Bytes alloced from RTOS: 8298496 bytes (7.91 Mb)

                     Chunks free: 15 bytes

       Number of mmapped regions: 40

  Total space in mmapped regions: 114208768 bytes (108.92 Mb)

           Total allocated space: 8153400 bytes (7.77 Mb)

           Total non-inuse space: 145096 bytes (141.69 Kb)

      Top-most, releasable space: 55416 bytes (54.11 Kb)

     Total allocated (incl mmap): 122507264 bytes (116.84 Mb)

          Total used (incl mmap): 122362168 bytes (116.70 Mb)

          Total free (incl mmap): 145096 bytes (141.69 Kb)

4 REPLIES
Cisco Employee

Re: WLC 5508 v6.0.202.0 reboot randomly.

Hello,

Let me begin by asking if there is a specific reason why the wlc is still running a deferred code?

Sent from Cisco Technical Support iPhone App

Cisco Employee

Re: WLC 5508 v6.0.202.0 reboot randomly.

I stand corrected. The code is still not deferred. I am still curios to find out why. The reason i am asking that is most likely you are hitting a bug which. The bug fix will be on different code and if you are upgrading then might as well upgrade to a more recent code train.

Sent from Cisco Technical Support iPhone App

Hall of Fame Super Gold

WLC 5508 v6.0.202.0 reboot randomly.

Contact Cisco and get the WLC RMA-ed.

Cisco Employee

WLC 5508 v6.0.202.0 reboot randomly.

I am not sure RMA is the best way to go about this.

controller crashes are usually fixed via code changes (aka bug fixes)

703
Views
0
Helpful
4
Replies
CreatePlease to create content