APs randomly will not join the controller

Unanswered Question
Apr 11th, 2014
User Badges:
  • Bronze, 100 points or more

Hi all,

In short, APs will join the controller immediately using the recovery image, but once it downloads 7.6.100 and reboots, it either cannot join the controller or will take up to 40 minutes.

 

The AP is in local mode. The AP 1040 eventually joined the controller, 40 minutes after we deleted all the files off the flash which is unacceptable.

 

*Apr  9 15:38:15.842: %CAPWAP-5-SENDJOIN: sending Join Request to 10.10.10.10

*Apr  9 15:38:15.851: %CAPWAP-3-ERRORLOG: Invalid event 10 & state 5 combination.

*Apr  9 15:38:15.851: %CAPWAP-3-ERRORLOG: CAPWAP SM handler: Failed to process message type 10 state 5.

*Apr  9 15:38:15.851: %CAPWAP-3-ERRORLOG: Failed to handle capwap control message from controller

*Apr  9 15:38:15.851: %CAPWAP-3-ERRORLOG: Failed to process encrypted capwap packet from 10.10.10.10

*Apr  9 15:38:16.304: %LINK-6-UPDOWN: Interface Dot11Radio0, changed state to down

*Apr  9 15:38:16.362: %LINK-5-CHANGED: Interface Dot11Radio0, changed state to reset

*Apr  9 15:38:16.427: %CAPWAP-5-JOINEDCONTROLLER: AP has joined controller CONTROLLER-DC

 

I also had our technician console into a 2602 and we were seeing exact same errors.

SUMMARY OF THIS CASE:

--AP is 1042 and wlc is on 7.6.100.0

--using dhcp option 43 but the AP has static ip

--checked the private config on the AP and it was not sending the discovery request to the desired  WLC

--cleared the private config

--primed the AP to desired WLC

--AP sending join request but receiving the following errors:

 

Apr  4 14:37:47.000: CAPWAP-3-ERRORLOG Go join a capwap controller

*Apr  4 14:36:47.000: CAPWAP-5-DTLSREQSEND DTLS connection request sent peer_ip: 10.10.10.11 peer_port: 5246

*Apr  4 14:36:47.169: DTLS_CLIENT_ERROR: ../capwap/base_capwap/dtls/base_capwap_dtls_handshake.c:90 First fragment for seq 2 is missing

*Apr  4 14:37:17.205: DTLS_CLIENT_ERROR: ../capwap/base_capwap/dtls/base_capwap_dtls_connection_db.c:2176 Max retransmission count reached!

--cleared the flash except rcv

--ap started downloading the image from the wlc but once it reboots the same errors appear again

--research made on the error message and apparently we are hitting a bug:

https://tools.cisco.com/bugsearch/bug/CSCul08933/?reffering_site=dumpcr

 

 

  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 5 (1 ratings)
Loading.
Leo Laohoo Fri, 04/11/2014 - 17:40
User Badges:
  • Super Gold, 25000 points or more
  • Hall of Fame,

    The Hall of Fame designation is a lifetime achievement award based on significant overall achievements in the community. 

  • Cisco Designated VIP,

    2017 LAN, Wireless

May I ask if you can post the output to the following commands: 

 

  1. WLC:  sh sysinfo; 
  2. WLC:  sh time; 
  3. AP:  sh version; 
  4. AP:  sh ip interface brief; and
  5. AP:  sh inventory
Abraham Camacho Tue, 04/15/2014 - 08:28
User Badges:
  • Bronze, 100 points or more

Hi Leo,

 

Next the information, I omitted some serial number and descriptions due to confidentiality matters.

 

(Cisco Controller) >show sysinfo

Manufacturer's Name.............................. Cisco Systems Inc.

Product Name..................................... Cisco Controller Product Version.................................. 7.6.100.0 Bootloader Version............................... 1.0.1 Field Recovery Image Version..................... 6.0.182.0 Firmware Version................................. FPGA 1.3, Env 1.6, USB console 1.27 Build Type....................................... DATA + WPS

System Name...................................... ABCDEF System Location..................................

System Contact...................................

System ObjectID.................................. 1.3.6.1.4.1.9.1.1069 Redundancy Mode.................................. Disabled IP Address....................................... XXXXXXX Last Reset....................................... Software reset System Up Time................................... 32 days 14 hrs 1 mins 41 secs System Timezone Location......................... (GMT -5:00) Eastern Time (US and Canada) System Stats Realtime Interval................... 5 System Stats Normal Interval..................... 180

 

Configured Country............................... CA  - Canada Operating Environment............................ Commercial (0 to 40 C)

Internal Temp Alarm Limits....................... 0 to 65 C Internal Temperature............................. +40 C External Temperature............................. +21 C Fan Status....................................... OK

State of 802.11b Network......................... Enabled State of 802.11a Network......................... Enabled Number of WLANs.................................. 5 Number of Active Clients......................... 2215

Burned-in MAC Address............................ F0:F7:55:2B:4A:80 Power Supply 1................................... Present, OK Power Supply 2................................... Absent Maximum number of APs supported.................. 500

(Cisco Controller) >show time

Time............................................. Mon Apr 14 08:40:08 2014

Timezone delta................................... 0:0 Timezone location................................ (GMT -5:00) Eastern Time (US and Canada)

NTP Servers

    NTP Polling Interval.........................     36000

     Index     NTP Key Index                  NTP Server                  NTP Msg Auth Status

    -------  ----------------------------------------------------------------------------------

       1              0                                              A.B.C.D                        AUTH DISABLED

 

AP>sh ver

Cisco IOS Software, C3500 Software (AP3G1-K9W8-M), Version 15.2(4)JB3, RELEASE SOFTWARE (fc1) Technical Support: http://www.cisco.com/techsupport Copyright (c) 1986-2013 by Cisco Systems, Inc.

Compiled Wed 18-Dec-13 21:36 by prod_rel_team

 

ROM: Bootstrap program is C3500 boot loader

BOOTLDR: C3500 Boot Loader (AP3G1-BOOT-M), Version 15.2 [chayan-apn-0411 116]

AP uptime is 2 days, 17 hours, 15 minutes System returned to ROM by reload System image file is "flash:/ap3g1-k9w8-mx.152-4.JB3/ap3g1-k9w8-xx.152-4.JB3"

Last reload reason:

 

cisco AIR-CAP3502I-A-K9 (PowerPC460exr) processor (revision A0) with 98294K/32768K bytes of memory.

Processor board ID XXXXXXXX

PowerPC460exr CPU at 666Mhz, revision number 0x18A8 Last reset from reload LWAPP image version 7.6.100.0

1 Gigabit Ethernet interface

2 802.11 Radios

 

32K bytes of flash-simulated non-volatile configuration memory.

Base ethernet MAC Address: 70:81:05:9E:F7:36

Part Number                          : 73-12175-05

PCA Assembly Number                  : 800-32268-05

PCA Revision Number                  : A0

PCB Serial Number                    : XXXXXXX

Top Assembly Part Number             : 800-32891-01

Top Assembly Serial Number           : FGL1534S7LP

Top Revision Number                  : A0

Product/Model Number                 : AIR-CAP3502I-A-K9  

 

 

 

Configuration register is 0xF

AP>sh ip int br

Interface                  IP-Address      OK? Method Status                Protocol

BVI1                       X.Y.W.Z                YES TFTP   up                    up     

Dot11Radio0                unassigned      NO  unset  up                    up     

Dot11Radio1                unassigned      NO  unset  up                    up     

GigabitEthernet0           unassigned      NO  unset  up                    up     

GigabitEthernet0.1         unassigned      YES unset  up                    up     

 

AP>sh inventory

NAME: "AP3500", DESCR: "Cisco Aironet 3500 Series (IEEE 802.11n) Access Point"

PID: AIR-CAP3502I-A-K9 , VID: V01, SN: XXXXXXX

Leo Laohoo Tue, 04/15/2014 - 22:53
User Badges:
  • Super Gold, 25000 points or more
  • Hall of Fame,

    The Hall of Fame designation is a lifetime achievement award based on significant overall achievements in the community. 

  • Cisco Designated VIP,

    2017 LAN, Wireless

I omitted some serial number

You omitted the WRONG serial number.  

Abraham Camacho Wed, 04/16/2014 - 04:21
User Badges:
  • Bronze, 100 points or more

Do you need that information?. If it so, what is the serial you want to check.

Let me take a look on the mesh ios you mentioned.

Leo Laohoo Tue, 04/15/2014 - 22:54
User Badges:
  • Super Gold, 25000 points or more
  • Hall of Fame,

    The Hall of Fame designation is a lifetime achievement award based on significant overall achievements in the community. 

  • Cisco Designated VIP,

    2017 LAN, Wireless

I've got a suspicion your AP is running MESH IOS.

Abraham Camacho Wed, 04/16/2014 - 08:25
User Badges:
  • Bronze, 100 points or more

Hi Leo,

 

What is the implication of using MESH IOS with respect to this issue?

 

thanks

Leo Laohoo Wed, 04/16/2014 - 17:32
User Badges:
  • Super Gold, 25000 points or more
  • Hall of Fame,

    The Hall of Fame designation is a lifetime achievement award based on significant overall achievements in the community. 

  • Cisco Designated VIP,

    2017 LAN, Wireless

If your AP is running mesh, then it means that you'll need to add the AP MAC address into the WLC database so the AP can join the controller.

Abraham Camacho Wed, 06/04/2014 - 15:27
User Badges:
  • Bronze, 100 points or more

As a general information, the problem was a DTLS bug found on version 7.6.100 which was addressed on an special software 7.6.100.25 that we installed recently and tested.

 

regards

Actions

This Discussion

 

 

Trending Topics: Other Wireless Mobility

client could not be authenticated
Network Analysis Module (NAM) Products
Cisco 6500 nam
reason 440 driver failure
Cisco password cracker
Cisco Wireless mode