12-14-2013 10:04 AM - edited 07-04-2021 01:25 AM
Hi,
I have recently been tasked with upgrading our old Autonomous APs to LWAPs. We have a 5508 WLC at our Virtual Co-Lo and I am using Flexconnect to accomadate local switching and dhcp at our sites. I have upgraded over 50 APs and joined them to the controller. These include only 1130AG and 1240AG models. However they are working flawlessly and staying connected to the controller. The issue I'm having is with a new batch of 2600 series APs staying connected to the controller. I have attempted to do research into what may be causing the disconnects but have yet to find a solution. I am using DNS to resolve the CAPWAP & LWAPP queries from the APs to the controller accross our WAN. In reading other posts I thought it may be an issue with packets getting dropped but have had our Vendor who manages Sonicwalls at both ends of the WAN confirm for me there is no packet loss. Below are logs I gathered using puttty from the AP & WLC. Any help would be greatly appreciated.
AP I'm doing the testing on:
NAME: "AP2600", DESCR: "Cisco Aironet 2600 Series (IEEE 802.11n) Access Point"
PID: AIR-CAP2602I-A-K9 , VID: V01, SN: FTX1740J8V1
WLC in question:
Manufacturer's Name.............................. Cisco Systems Inc.
Product Name..................................... Cisco Controller
Product Version.................................. 7.3.112.0
Bootloader Version............................... 1.0.1
Field Recovery Image Version..................... 6.0.182.0
Firmware Version................................. FPGA 1.3, Env 1.6, USB console 1.27
Build Type....................................... DATA + WPS
System Name...................................... wificontroller
System Location.................................. Corp
System Contact................................... Net Engineer
System ObjectID.................................. 1.3.6.1.4.1.9.1.1069
Redundancy Mode.................................. Disabled
IP Address....................................... 10.250.32.8
Last Reset....................................... Software reset
System Up Time................................... 190 days 3 hrs 34 mins 24 secs
System Timezone Location......................... (GMT -5:00) Eastern Time (US and Canada)
Configured Country............................... US - United States
Operating Environment............................ Commercial (0 to 40 C)
Internal Temp Alarm Limits....................... 0 to 65 C
--More-- or (q)uit
Internal Temperature............................. +38 C
External Temperature............................. +20 C
Fan Status....................................... OK
State of 802.11b Network......................... Enabled
State of 802.11a Network......................... Enabled
Number of WLANs.................................. 14
Number of Active Clients......................... 71
Burned-in MAC Address............................ C8:9C:1D:8C:52:E0
Power Supply 1................................... Present, OK
Power Supply 2................................... Absent
Maximum number of APs supported.................. 100
Here is the output that keeps on occuring as the AP joins the WLC for a brief time and then changes to standalone mode
WT-4thFlr-AP3#
*Dec 14 15:42:04.419: %CAPWAP-3-ERRORLOG: Retransmission count for packet exceeded max(CAPWAP_WTP_EVENT_REQUEST
., 3)
*Dec 14 15:42:11.443: %EVT-4-WRN: Write of flash:/event.capwap done
*Dec 14 15:42:11.483: %LWAPP-3-CLIENTERRORLOG: Switching to Standalone mode
*Dec 14 15:42:11.487: %CAPWAP-3-ERRORLOG: GOING BACK TO DISCOVER MODE
*Dec 14 15:42:11.487: %DTLS-5-SEND_ALERT: Send FATAL : Close notify Alert to 10.250.32.8:5246
*Dec 14 15:42:11.571: %WIDS-6-DISABLED: IDS Signature is removed and disabled.
*Dec 14 15:42:21.575: %CAPWAP-3-ERRORLOG: Selected MWAR 'wificontroller'(index 0).
*Dec 14 15:42:21.575: %CAPWAP-3-ERRORLOG: Go join a capwap controller
*Dec 14 15:42:12.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: 10.250.32.8 peer_port: 5246
*Dec 14 15:42:14.303: %CAPWAP-5-DTLSREQSUCC: DTLS connection created sucessfully peer_ip: 10.250.32.8 peer_port: 5246
*Dec 14 15:42:14.303: %CAPWAP-5-SENDJOIN: sending Join Request to 10.250.32.8
*Dec 14 15:42:15.127: Starting Ethernet promiscuous mode
*Dec 14 15:42:15.535: %LWAPP-4-CLIENTEVENTLOG: OfficeExtend Localssid saved in AP flash
*Dec 14 15:42:15.667: ac_first_hop_mac - IP:10.1.2.250 Hop IP:10.1.2.250 IDB:BVI1
*Dec 14 15:42:15.667: Setting AC first hop MAC: 0017.c575.a23c
*Dec 14 15:42:15.855: %CAPWAP-5-JOINEDCONTROLLER: AP has joined controller wificontroller
*Dec 14 15:42:15.911: %LWAPP-4-CLIENTEVENTLOG: No Flex ACL map configuration file to load. Connect to controller to get configuration file
*Dec 14 15:42:15.911: %LWAPP-4-CLIENTEVENTLOG: No Flex ACL map configuration file to load. Connect to controller to get configuration file
*Dec 14 15:42:15.911: %LWAPP-4-CLIENTEVENTLOG: No LS Flex ACL map configuration file to load. Connect to controller to get configuration file
*Dec 14 15:42:15.915: %LWAPP-4-CLIENTEVENTLOG: No Central Dhcp map configuration file to load. Connect to controller to get configuration file
*Dec 14 15:42:15.915: %LWAPP-3-CLIENTERRORLOG: Switching to Connected mode
*Dec 14 15:42:23.639: %WIDS-6-ENABLED: IDS Signature is loaded and enabled
*Dec 14 15:42:34.615: %CLEANAIR-6-STATE: Slot 0 disabled
*Dec 14 15:42:34.615: %CLEANAIR-6-STATE: Slot 1 disabled
*Dec 14 15:45:43.783: %CAPWAP-3-ERRORLOG: Retransmission count for packet exceeded max(CAPWAP_WTP_EVENT_REQUEST
., 11)
*Dec 14 15:45:43.787: %LWAPP-3-CLIENTERRORLOG: Switching to Standalone mode
*Dec 14 15:45:43.787: %CAPWAP-3-ERRORLOG: GOING BACK TO DISCOVER MODE
*Dec 14 15:45:43.787: %DTLS-5-SEND_ALERT: Send FATAL : Close notify Alert to 10.250.32.8:5246
*Dec 14 15:45:43.867: %WIDS-6-DISABLED: IDS Signature is removed and disabled.
*Dec 14 15:45:53.867: %CAPWAP-3-ERRORLOG: Selected MWAR 'wificontroller'(index 0).
*Dec 14 15:45:53.867: %CAPWAP-3-ERRORLOG: Go join a capwap controller
*Dec 14 15:45:44.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: 10.250.32.8 peer_port: 5246
*Dec 14 15:45:46.315: %CAPWAP-5-DTLSREQSUCC: DTLS connection created sucessfully peer_ip: 10.250.32.8 peer_port: 5246
*Dec 14 15:45:46.315: %CAPWAP-5-SENDJOIN: sending Join Request to 10.250.32.8
*Dec 14 15:45:46.487: Starting Ethernet promiscuous mode
*Dec 14 15:45:49.903: %LWAPP-4-CLIENTEVENTLOG: OfficeExtend Localssid saved in AP flash
*Dec 14 15:45:50.031: ac_first_hop_mac - IP:10.1.2.250 Hop IP:10.1.2.250 IDB:BVI1
*Dec 14 15:45:50.031: Setting AC first hop MAC: 0017.c575.a23c
Here are the results of debug capwap client event on the AP:
WT-4thFlr-AP3#debug capwap client event
CAPWAP Client EVENT display debugging is on
WT-4thFlr-AP3#
*Dec 14 15:54:58.335: %CAPWAP-3-EVENTLOG: Echo Interval Expired.
*Dec 14 15:54:58.335: %CAPWAP-3-EVENTLOG: Sending packet to AC
*Dec 14 15:54:58.335: %CAPWAP-3-EVENTLOG: Echo Request sent to 10.250.32.8
*Dec 14 15:54:58.343: %CAPWAP-3-EVENTLOG: Resetting reTransmissionCnt to 0
*Dec 14 15:54:58.343: %CAPWAP-3-EVENTLOG: Queue Empty.
*Dec 14 15:54:58.343: %CAPWAP-3-EVENTLOG: Echo Response from 10.250.32.8
*Dec 14 15:55:08.000: %CAPWAP-3-EVENTLOG: Setting time to 15:55:08 UTC Dec 14 2013
*Dec 14 15:55:25.579: %CAPWAP-3-EVENTLOG: Sending packet to AC
*Dec 14 15:55:25.587: %CAPWAP-3-EVENTLOG: Resetting reTransmissionCnt to 0
*Dec 14 15:55:25.587: %CAPWAP-3-EVENTLOG: Queue Empty.
*Dec 14 15:55:25.587: %CAPWAP-3-EVENTLOG: Wtp Event Response from 10.250.32.8
*Dec 14 15:55:25.827: %CAPWAP-3-EVENTLOG: Sending packet to AC
*Dec 14 15:55:25.835: %CAPWAP-3-EVENTLOG: Resetting reTransmissionCnt to 0
*Dec 14 15:55:25.835: %CAPWAP-3-EVENTLOG: Queue Empty.
*Dec 14 15:55:25.835: %CAPWAP-3-EVENTLOG: Wtp Event Response from 10.250.32.8
*Dec 14 15:55:55.835: %CAPWAP-3-EVENTLOG: Echo Interval Expired.
*Dec 14 15:55:55.835: %CAPWAP-3-EVENTLOG: Sending packet to AC
*Dec 14 15:55:55.835: %CAPWAP-3-EVENTLOG: Echo Request sent to 10.250.32.8
*Dec 14 15:55:55.843: %CAPWAP-3-EVENTLOG: Resetting reTransmissionCnt to 0
*Dec 14 15:55:55.843: %CAPWAP-3-EVENTLOG: Queue Empty.
*Dec 14 15:55:55.843: %CAPWAP-3-EVENTLOG: Echo Response from 10.250.32.8
*Dec 14 15:55:56.000: %CAPWAP-3-EVENTLOG: Setting time to 15:55:56 UTC Dec 14 2013
*Dec 14 15:56:25.735: %CAPWAP-3-EVENTLOG: Sending packet to AC
*Dec 14 15:56:25.743: %CAPWAP-3-EVENTLOG: Resetting reTransmissionCnt to 0
*Dec 14 15:56:25.743: %CAPWAP-3-EVENTLOG: Queue Empty.
*Dec 14 15:56:25.743: %CAPWAP-3-EVENTLOG: Wtp Event Response from 10.250.32.8
*Dec 14 15:56:25.983: %CAPWAP-3-EVENTLOG: Sending packet to AC
*Dec 14 15:56:25.991: %CAPWAP-3-EVENTLOG: Resetting reTransmissionCnt to 0
*Dec 14 15:56:25.991: %CAPWAP-3-EVENTLOG: Queue Empty.
*Dec 14 15:56:25.991: %CAPWAP-3-EVENTLOG: Wtp Event Response from 10.250.32.8
*Dec 14 15:56:55.991: %CAPWAP-3-EVENTLOG: Echo Interval Expired.
*Dec 14 15:56:55.991: %CAPWAP-3-EVENTLOG: Sending packet to AC
*Dec 14 15:56:55.991: %CAPWAP-3-EVENTLOG: Echo Request sent to 10.250.32.8
*Dec 14 15:56:55.999: %CAPWAP-3-EVENTLOG: Resetting reTransmissionCnt to 0
*Dec 14 15:56:55.999: %CAPWAP-3-EVENTLOG: Queue Empty.
*Dec 14 15:56:55.999: %CAPWAP-3-EVENTLOG: Echo Response from 10.250.32.8
*Dec 14 15:56:56.000: %CAPWAP-3-EVENTLOG: Setting time to 15:56:56 UTC Dec 14 2013
Here are the results of debug capwap client packet detail:
WT-4thFlr-AP3#
*Dec 14 15:59:01.823: <<<< Start of CAPWAP Packet >>>>
*Dec 14 15:59:01.823: CAPWAP Control mesg Sent to 10.250.32.8, Port 5246
*Dec 14 15:59:01.823: Msg Type : CAPWAP_ECHO_REQUEST
*Dec 14 15:59:01.823: Msg Length : 0
*Dec 14 15:59:01.823: Msg SeqNum : 44
*Dec 14 15:59:01.823: <<<< End of CAPWAP Packet >>>>
*Dec 14 15:59:01.831: <<<< Start of CAPWAP Packet >>>>
*Dec 14 15:59:01.831: CAPWAP Control mesg Recd from 10.250.32.8, Port 5246
*Dec 14 15:59:01.831: HLEN 2, Radio ID 0, WBID 1
*Dec 14 15:59:01.831: Msg Type : CAPWAP_ECHO_RESPONSE
*Dec 14 15:59:01.831: Msg Length : 15
*Dec 14 15:59:01.831: Msg SeqNum : 44
*Dec 14 15:59:01.831:
*Dec 14 15:59:01.831: Type : CAPWAP_MSGELE_VENDOR_SPECIFIC_PAYLOAD, Length 11
*Dec 14 15:59:01.831: Vendor Identifier : 0x00409600
*Dec 14 15:59:01.831:
*Dec 14 15:59:01.831:
IE : UNKNOWN IE 151
*Dec 14 15:59:01.831: IE Length : 5
*Dec 14 15:59:01.831: Decode routine not available, Printing Hex Dump
*Dec 14 15:59:01.831:
52 AC 80 46 00
*Dec 14 15:59:01.831: <<<< End of CAPWAP Packet >>>>
*Dec 14 15:59:20.931: <<<< Start of CAPWAP Packet >>>>
*Dec 14 15:59:20.931: CAPWAP Control mesg Recd from 10.250.32.8, Port 5246
*Dec 14 15:59:20.931: HLEN 2, Radio ID 0, WBID 1
*Dec 14 15:59:20.931: Msg Type : CAPWAP_CONFIGURATION_UPDATE_REQUEST
*Dec 14 15:59:20.931: Msg Length : 93
*Dec 14 15:59:20.931: Msg SeqNum : 38
*Dec 14 15:59:20.931:
*Dec 14 15:59:20.931: Type : CAPWAP_MSGELE_VENDOR_SPECIFIC_PAYLOAD, Length 89
*Dec 14 15:59:20.931: Vendor Identifier : 0x00409600
*Dec 14 15:59:20.931:
*Dec 14 15:59:20.931:
IE : RRM_NEIGHBOR_CTRL_PAYLOAD
*Dec 14 15:59:20.931: IE Length : 83
*Dec 14 15:59:20.931: Decode routine not available, Printing Hex Dump
*Dec 14 15:59:20.931:
00 0A FA 20 08 01 F4 00 07 0A FA 20 08 03 00 01
01 00 3C 00 B4 2E 06 2E E7 B4 94 51 B2 C7 79 25
22 FD BE 04 F6 00 00 00 00 00 00 00 00 4F 50 52
53 2D 57 69 46 69 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 01 06 0B
01 01 01
*Dec 14 15:59:20.931: <<<< End of CAPWAP Packet >>>>
*Dec 14 15:59:20.931: <<<< Start of CAPWAP Packet >>>>
*Dec 14 15:59:20.931: CAPWAP Control mesg Sent to 10.250.32.8, Port 5246
*Dec 14 15:59:20.931: Msg Type : CAPWAP_CONFIGURATION_UPDATE_RESPONSE
*Dec 14 15:59:20.931: Msg Length : 8
*Dec 14 15:59:20.931: Msg SeqNum : 38
*Dec 14 15:59:20.931:
*Dec 14 15:59:20.931: Type : CAPWAP_MSGELE_RESULT_CODE, Length 4
*Dec 14 15:59:20.931: Result Code : CAPWAP_SUCCESS
*Dec 14 15:59:20.931: <<<< End of CAPWAP Packet >>>>
*Dec 14 15:59:21.139: <<<< Start of CAPWAP Packet >>>>
*Dec 14 15:59:21.139: CAPWAP Control mesg Recd from 10.250.32.8, Port 5246
*Dec 14 15:59:21.139: HLEN 2, Radio ID 0, WBID 1
*Dec 14 15:59:21.139: Msg Type : CAPWAP_CONFIGURATION_UPDATE_REQUEST
*Dec 14 15:59:21.139: Msg Length : 111
*Dec 14 15:59:21.139: Msg SeqNum : 39
*Dec 14 15:59:21.139:
*Dec 14 15:59:21.139: Type : CAPWAP_MSGELE_VENDOR_SPECIFIC_PAYLOAD, Length 107
*Dec 14 15:59:21.139: Vendor Identifier : 0x00409600
*Dec 14 15:59:21.139:
*Dec 14 15:59:21.139:
IE : RRM_NEIGHBOR_CTRL_PAYLOAD
*Dec 14 15:59:21.139: IE Length : 101
*Dec 14 15:59:21.139: Decode routine not available, Printing Hex Dump
*Dec 14 15:59:21.143:
01 0A FA 20 08 01 F4 00 07 0A FA 20 08 0C 00 01
01 00 3C 00 B4 2E 06 2E E7 B4 94 51 B2 C7 79 25
22 FD BE 04 F6 00 00 00 00 00 00 00 00 4F 50 52
53 2D 57 69 46 69 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 24 28 2C
30 34 38 3C 40 95 99 9D A1 01 01 01 01 01 01 01
01 01 01 01 01
*Dec 14 15:59:21.143: <<<< End of CAPWAP Packet >>>>
*Dec 14 15:59:21.143: <<<< Start of CAPWAP Packet >>>>
*Dec 14 15:59:21.143: CAPWAP Control mesg Sent to 10.250.32.8, Port 5246
*Dec 14 15:59:21.143: Msg Type : CAPWAP_CONFIGURATION_UPDATE_RESPONSE
*Dec 14 15:59:21.143: Msg Length : 8
*Dec 14 15:59:21.143: Msg SeqNum : 39
*Dec 14 15:59:21.143:
*Dec 14 15:59:21.143: Type : CAPWAP_MSGELE_RESULT_CODE, Length 4
*Dec 14 15:59:21.143: Result Code : CAPWAP_SUCCESS
*Dec 14 15:59:21.143: <<<< End of CAPWAP Packet >>>>
*Dec 14 15:59:25.547: <<<< Start of CAPWAP Packet >>>>
*Dec 14 15:59:25.547: CAPWAP Control mesg Sent to 10.250.32.8, Port 5246
*Dec 14 15:59:25.547: Msg Type : CAPWAP_WTP_EVENT_REQUEST
*Dec 14 15:59:25.547: Msg Length : 14
*Dec 14 15:59:25.547: Msg SeqNum : 45
*Dec 14 15:59:25.547:
*Dec 14 15:59:25.547: Type : CAPWAP_MSGELE_VENDOR_SPECIFIC_PAYLOAD, Length 10
*Dec 14 15:59:25.547: Vendor Identifier : 0x00409600
*Dec 14 15:59:25.547:
*Dec 14 15:59:25.547:
IE : RRM_LOAD_DATA_PAYLOAD
*Dec 14 15:59:25.547: IE Length : 4
*Dec 14 15:59:25.547: slot 0 rxLoad 0 txLoad 0 ccaLoad 33
*Dec 14 15:59:25.547: <<<< End of CAPWAP Packet >>>>
*Dec 14 15:59:25.555: <<<< Start of CAPWAP Packet >>>>
*Dec 14 15:59:25.555: CAPWAP Control mesg Recd from 10.250.32.8, Port 5246
*Dec 14 15:59:25.555: HLEN 2, Radio ID 0, WBID 1
*Dec 14 15:59:25.555: Msg Type : CAPWAP_WTP_EVENT_RESPONSE
*Dec 14 15:59:25.555: Msg Length : 0
*Dec 14 15:59:25.555: Msg SeqNum : 45
*Dec 14 15:59:25.555: <<<< End of CAPWAP Packet >>>>
*Dec 14 15:59:25.795: <<<< Start of CAPWAP Packet >>>>
*Dec 14 15:59:25.795: CAPWAP Control mesg Sent to 10.250.32.8, Port 5246
*Dec 14 15:59:25.795: Msg Type : CAPWAP_WTP_EVENT_REQUEST
*Dec 14 15:59:25.795: Msg Length : 14
*Dec 14 15:59:25.795: Msg SeqNum : 46
*Dec 14 15:59:25.795:
*Dec 14 15:59:25.795: Type : CAPWAP_MSGELE_VENDOR_SPECIFIC_PAYLOAD, Length 10
*Dec 14 15:59:25.795: Vendor Identifier : 0x00409600
*Dec 14 15:59:25.795:
*Dec 14 15:59:25.795:
IE : RRM_LOAD_DATA_PAYLOAD
*Dec 14 15:59:25.795: IE Length : 4
*Dec 14 15:59:25.795: slot 1 rxLoad 0 txLoad 0 ccaLoad 0
*Dec 14 15:59:25.795: <<<< End of CAPWAP Packet >>>>
*Dec 14 15:59:25.803: <<<< Start of CAPWAP Packet >>>>
*Dec 14 15:59:25.803: CAPWAP Control mesg Recd from 10.250.32.8, Port 5246
*Dec 14 15:59:25.803: HLEN 2, Radio ID 0, WBID 1
*Dec 14 15:59:25.803: Msg Type : CAPWAP_WTP_EVENT_RESPONSE
*Dec 14 15:59:25.803: Msg Length : 0
*Dec 14 15:59:25.803: Msg SeqNum : 46
*Dec 14 15:59:25.803: <<<< End of CAPWAP Packet >>>>
*Dec 14 15:59:30.375: <<<< Start of CAPWAP Packet >>>>
*Dec 14 15:59:30.375: CAPWAP Control mesg Recd from 10.250.32.8, Port 5246
*Dec 14 15:59:30.375: HLEN 2, Radio ID 0, WBID 1
*Dec 14 15:59:30.375: Msg Type : CAPWAP_CONFIGURATION_UPDATE_REQUEST
*Dec 14 15:59:30.375: Msg Length : 17
*Dec 14 15:59:30.375: Msg SeqNum : 40
*Dec 14 15:59:30.375:
*Dec 14 15:59:30.375: Type : CAPWAP_MSGELE_VENDOR_SPECIFIC_PAYLOAD, Length 13
*Dec 14 15:59:30.375: Vendor Identifier : 0x00409600
SlotId : 0
Mobile Mac Addr : BC:52:B7:E3:17:CB
*Dec 14 15:59:30.375: <<<< End of CAPWAP Packet >>>>
*Dec 14 15:59:30.375: <<<< Start of CAPWAP Packet >>>>
*Dec 14 15:59:30.375: CAPWAP Control mesg Sent to 10.250.32.8, Port 5246
*Dec 14 15:59:30.375: Msg Type : CAPWAP_CONFIGURATION_UPDATE_RESPONSE
*Dec 14 15:59:30.379: Msg Length : 8
*Dec 14 15:59:30.379: Msg SeqNum : 40
*Dec 14 15:59:30.379:
*Dec 14 15:59:30.379: Type : CAPWAP_MSGELE_RESULT_CODE, Length 4
*Dec 14 15:59:30.379: Result Code : CAPWAP_SUCCESS
*Dec 14 15:59:30.379: <<<< End of CAPWAP Packet >>>>
*Dec 14 15:59:30.387: <<<< Start of CAPWAP Packet >>>>
*Dec 14 15:59:30.387: CAPWAP Control mesg Recd from 10.250.32.8, Port 5246
*Dec 14 15:59:30.387: HLEN 2, Radio ID 0, WBID 1
*Dec 14 15:59:30.387: Msg Type : CAPWAP_WTP_EVENT_RESPONSE
*Dec 14 15:59:30.387: Msg Length : 0
*Dec 14 15:59:30.387: Msg SeqNum : 47
*Dec 14 15:59:30.387: <<<< End of CAPWAP Packet >>>>
*Dec 14 16:00:00.387: <<<< Start of CAPWAP Packet >>>>
*Dec 14 16:00:00.387: CAPWAP Control mesg Sent to 10.250.32.8, Port 5246
*Dec 14 16:00:00.387: Msg Type : CAPWAP_ECHO_REQUEST
*Dec 14 16:00:00.387: Msg Length : 0
*Dec 14 16:00:00.387: Msg SeqNum : 48
*Dec 14 16:00:00.387: <<<< End of CAPWAP Packet >>>>
*Dec 14 16:00:00.395: <<<< Start of CAPWAP Packet >>>>
*Dec 14 16:00:00.395: CAPWAP Control mesg Recd from 10.250.32.8, Port 5246
*Dec 14 16:00:00.395: HLEN 2, Radio ID 0, WBID 1
*Dec 14 16:00:00.395: Msg Type : CAPWAP_ECHO_RESPONSE
*Dec 14 16:00:00.395: Msg Length : 15
*Dec 14 16:00:00.395: Msg SeqNum : 48
*Dec 14 16:00:00.395:
*Dec 14 16:00:00.395: Type : CAPWAP_MSGELE_VENDOR_SPECIFIC_PAYLOAD, Length 11
*Dec 14 16:00:00.395: Vendor Identifier : 0x00409600
*Dec 14 16:00:00.395:
*Dec 14 16:00:00.395:
IE : UNKNOWN IE 151
*Dec 14 16:00:00.395: IE Length : 5
*Dec 14 16:00:00.395: Decode routine not available, Printing Hex Dump
*Dec 14 16:00:00.395:
52 AC 80 81 00
*Dec 14 16:00:00.395: <<<< End of CAPWAP Packet >>>>
Solved! Go to Solution.
12-19-2013 04:20 AM
try reducing mtu on ap 2600.
12-14-2013 05:07 PM
Since your WLC is set up for FlexConnect, can you please check your AP Policies and see if the 2600 is in the list or not?
12-14-2013 05:50 PM
Under my AP Policies I only have "Accept Manufactured Installed Certificate (MIC)" checked. I attempted to add the AP based on MAC Address (c0:67:af:6f:25:70) with this certificate type but still have the same issue. I then ran the following debug on my controller and this is the output I recieve regarding that MAC. I tried to cut the output short because it get's somewhat redundant but was unsure what exactly to look for in the output. Should I be selecting a different certificate type? I am somewhat new to wireless technologies but doing my best to pick things up so if this seems trivial please forgive my ignorance.
debug pm pki enable
*sshpmLscTask: Dec 14 20:42:56.450: sshpmLscTask: LSC Task received a message 4
*spamApTask6: Dec 14 20:42:58.840: sshpmGetIssuerHandles: locking ca cert table
*spamApTask6: Dec 14 20:42:58.841: sshpmGetIssuerHandles: calling x509_alloc() for user cert
*spamApTask6: Dec 14 20:42:58.841: sshpmGetIssuerHandles: calling x509_decode()
*spamApTask6: Dec 14 20:42:58.845: sshpmGetIssuerHandles:
*spamApTask6: Dec 14 20:42:58.845: sshpmGetIssuerHandles:
*spamApTask6: Dec 14 20:42:58.845: sshpmGetIssuerHandles: Mac Address in subject is c0:67:af:6f:25:70
*spamApTask6: Dec 14 20:42:58.845: sshpmGetIssuerHandles: Cert Name in subject is AP3G2-c067af6f2570
*spamApTask6: Dec 14 20:42:58.845: sshpmGetIssuerHandles: Cert is issued by Cisco Systems.
*spamApTask6: Dec 14 20:42:58.845: sshpmGetCID: called to evaluate
*spamApTask6: Dec 14 20:42:58.845: sshpmGetCID: comparing to row 0, CA cert >bsnOldDefaultCaCert<
*spamApTask6: Dec 14 20:42:58.845: sshpmGetCID: comparing to row 1, CA cert >bsnDefaultRootCaCert<
*spamApTask6: Dec 14 20:42:58.845: sshpmGetCID: comparing to row 2, CA cert >bsnDefaultCaCert<
*spamApTask6: Dec 14 20:42:58.845: sshpmGetCID: comparing to row 3, CA cert >bsnDefaultBuildCert<
*spamApTask6: Dec 14 20:42:58.845: sshpmGetCID: comparing to row 4, CA cert >cscoDefaultNewRootCaCert<
*spamApTask6: Dec 14 20:42:58.845: sshpmGetCID: comparing to row 5, CA cert >cscoDefaultMfgCaCert<
*spamApTask6: Dec 14 20:42:58.845: sshpmGetCertFromCID: called to get cert for CID 282aef7e
*spamApTask6: Dec 14 20:42:58.845: sshpmGetCertFromCID: comparing to row 0, certname >bsnOldDefaultCaCert<
*spamApTask6: Dec 14 20:42:58.845: sshpmGetCertFromCID: comparing to row 1, certname >bsnDefaultRootCaCert<
*spamApTask6: Dec 14 20:42:58.845: sshpmGetCertFromCID: comparing to row 2, certname >bsnDefaultCaCert<
*spamApTask6: Dec 14 20:42:58.845: sshpmGetCertFromCID: comparing to row 3, certname >bsnDefaultBuildCert<
*spamApTask6: Dec 14 20:42:58.845: sshpmGetCertFromCID: comparing to row 4, certname >cscoDefaultNewRootCaCert<
*spamApTask6: Dec 14 20:42:58.845: sshpmGetCertFromCID: comparing to row 5, certname >cscoDefaultMfgCaCert<
*spamApTask6: Dec 14 20:42:58.845: ssphmUserCertVerify: calling x509_decode()
*spamApTask6: Dec 14 20:42:58.856: ssphmUserCertVerify: user cert verfied using >cscoDefaultMfgCaCert<
*spamApTask6: Dec 14 20:42:58.856: sshpmGetIssuerHandles: ValidityString (current): 2013/12/15/01:42:58
*spamApTask6: Dec 14 20:42:58.856: sshpmGetIssuerHandles: ValidityString (NotBefore): 2013/08/25/13:01:22
*spamApTask6: Dec 14 20:42:58.856: sshpmGetIssuerHandles: ValidityString (NotAfter): 2023/08/25/13:11:22
*spamApTask6: Dec 14 20:42:58.856: sshpmGetIssuerHandles: getting cisco ID cert handle...
*spamApTask6: Dec 14 20:42:58.856: sshpmGetCID: called to evaluate
*spamApTask6: Dec 14 20:42:58.856: sshpmGetCID: comparing to row 0, CA cert >bsnOldDefaultCaCert<
*spamApTask6: Dec 14 20:42:58.856: sshpmGetCID: comparing to row 1, CA cert >bsnDefaultRootCaCert<
*spamApTask6: Dec 14 20:42:58.856: sshpmGetCID: comparing to row 2, CA cert >bsnDefaultCaCert<
*spamApTask6: Dec 14 20:42:58.856: sshpmGetCID: comparing to row 3, CA cert >bsnDefaultBuildCert<
*spamApTask6: Dec 14 20:42:58.856: sshpmGetCID: comparing to row 4, CA cert >cscoDefaultNewRootCaCert<
*spamApTask6: Dec 14 20:42:58.856: sshpmGetCID: comparing to row 5, CA cert >cscoDefaultMfgCaCert<
*spamApTask6: Dec 14 20:42:58.856: sshpmGetCID: comparing to row 0, ID cert >bsnOldDefaultIdCert<
*spamApTask6: Dec 14 20:42:58.856: sshpmGetCID: comparing to row 1, ID cert >bsnDefaultIdCert<
*spamApTask6: Dec 14 20:42:58.856: sshpmGetCID: comparing to row 2, ID cert >cscoDefaultIdCert<
*spamApTask6: Dec 14 20:42:58.857: sshpmFreePublicKeyHandle: called with 0x2c5f0cb8
*spamApTask6: Dec 14 20:42:58.857: sshpmFreePublicKeyHandle: freeing public key
*spamApTask0: Dec 14 20:43:17.451: sshpmGetCID: called to evaluate
*spamApTask0: Dec 14 20:43:17.451: sshpmGetCID: comparing to row 0, CA cert >bsnOldDefaultCaCert<
*spamApTask0: Dec 14 20:43:17.451: sshpmGetCID: comparing to row 1, CA cert >bsnDefaultRootCaCert<
*spamApTask0: Dec 14 20:43:17.451: sshpmGetCID: comparing to row 2, CA cert >bsnDefaultCaCert<
*spamApTask0: Dec 14 20:43:17.451: sshpmGetCID: comparing to row 3, CA cert >bsnDefaultBuildCert<
*spamApTask0: Dec 14 20:43:17.451: sshpmGetCID: comparing to row 4, CA cert >cscoDefaultNewRootCaCert<
*spamApTask0: Dec 14 20:43:17.451: sshpmGetCID: comparing to row 5, CA cert >cscoDefaultMfgCaCert<
*spamApTask0: Dec 14 20:43:17.451: sshpmGetCID: comparing to row 0, ID cert >bsnOldDefaultIdCert<
*spamApTask0: Dec 14 20:43:17.451: sshpmGetCID: comparing to row 1, ID cert >bsnDefaultIdCert<
*spamApTask0: Dec 14 20:43:17.451: sshpmGetCID: comparing to row 2, ID cert >cscoDefaultIdCert<
*spamApTask0: Dec 14 20:43:17.451: sshpmGetCertFromCID: called to get cert for CID 183fd2b6
*spamApTask0: Dec 14 20:43:17.451: sshpmGetCertFromCID: comparing to row 0, certname >bsnOldDefaultCaCert<
*spamApTask0: Dec 14 20:43:17.451: sshpmGetCertFromCID: comparing to row 1, certname >bsnDefaultRootCaCert<
*spamApTask0: Dec 14 20:43:17.451: sshpmGetCertFromCID: comparing to row 2, certname >bsnDefaultCaCert<
*spamApTask0: Dec 14 20:43:17.451: sshpmGetCertFromCID: comparing to row 3, certname >bsnDefaultBuildCert<
*spamApTask0: Dec 14 20:43:17.451: sshpmGetCertFromCID: comparing to row 4, certname >cscoDefaultNewRootCaCert<
*spamApTask0: Dec 14 20:43:17.451: sshpmGetCertFromCID: comparing to row 5, certname >cscoDefaultMfgCaCert<
*spamApTask0: Dec 14 20:43:17.451: sshpmGetCertFromCID: comparing to row 0, certname >bsnOldDefaultIdCert<
*spamApTask0: Dec 14 20:43:17.451: sshpmGetCertFromCID: comparing to row 1, certname >bsnDefaultIdCert<
*spamApTask0: Dec 14 20:43:17.451: sshpmGetCertFromCID: comparing to row 2, certname >cscoDefaultIdCert<
*spamApTask0: Dec 14 20:43:17.451: sshpmGetCID: called to evaluate
12-14-2013 06:20 PM
*Dec 14 15:45:49.903: %LWAPP-4-CLIENTEVENTLOG: OfficeExtend Localssid saved in AP flash
If the AP is currently in the same location as the WLC, disable FlexConnect and switch the AP into local. See what happens.
12-14-2013 06:25 PM
The AP is at one of our nursing facilities and the WLC is at our Co-Lo in another building connected via WAN. We only have the one controller and this AP came straight out of a box so has not been associated with any other controllers. Any other suggestions or useful debugs I could try to determine the problem?
12-15-2013 05:39 AM
I'm not sure if this is helpful but here is the output I recieved from the following show command. The high number of link failures is what stands out to me but still can't determine what is causing this.
WT-4thFlr-AP3#term mon
WT-4thFlr-AP3#show capwap client config
configMagicMark 0xF1E2D3C4
chkSumV2 27340
chkSumV1 10268
swVer 7.3.112.0
adminState ADMIN_ENABLED(1)
name WT-4thFlr-AP3
location WT 4th Floor AP3
group name WestminsterThurber
mwarName wificontroller
mwarIPAddress 10.250.32.8
mwarName
mwarIPAddress 0.0.0.0
mwarName
mwarIPAddress 0.0.0.0
ssh status Enabled
Telnet status Disabled
numOfSlots 2
spamRebootOnAssert 1
spamStatTimer 180
randSeed 0x2082
transport SPAM_TRANSPORT_L3(2)
transportCfg SPAM_TRANSPORT_DEFAULT(0)
initialisation SPAM_PRODUCTION_DISCOVERY(1)
--More-- ApMode FlexConnect
ApSubMode Not Configured
AP Rogue Detection Mode Enabled
OfficeExtend AP [0] Disabled
OfficeExtend AP JoinMode[0] Standard
Discovery Timer 10 secs
Heart Beat Timer 30 secs
Led State Enabled 1
Primed Interval 0
AP ILP Pre-Standard Switch Support Disabled
AP Power Injector Disabled
Infrastructure MFP validation Disabled
Configured Switch 1 Addr 10.250.32.8
non-occupancy channels:
Ethernet (Duplex/Speed) auto/auto
Slot 0
adminstate ADMIN_ENABLED(1)
radioType RADIO_TYPE_80211bg
CleanAirAdminState Enabled
countryCode US
countryISOCode US
chanAutoCfg CONFIG_AUTO
--More-- channel 6
channel width 20
extension channel none
txPowerAutoCfg CONFIG_AUTO
txPowerLevel 2
diversitySelection DIVERSITY_ENABLED
htRxAntennaSelection 0F
htTxAntennaSelection 0F
beamformCfg 03
Antenna Mode ANTENNA_OMNI
antennaSelection_0 INTERNAL_ANTENNA
antennaSelection_1 INTERNAL_ANTENNA
antennaSelection_2 INTERNAL_ANTENNA
antennaSelection_3 INTERNAL_ANTENNA
twiceExtAntennaGain 0
Profile Mode CONFIG_AUTO
Load Profile
rfBusyThreshold 0
numClientsThreshold 0
bytesPerSecThreshold 0
Interference Profile
InterferenceThreshold 0
Noise Profile
--More-- NoiseThreshold 0
Coverage Profile
SNRThreshold 0
ExceptionThreshold 0
minClientsThreshold 0
11gSupport Enabled
override mode Disabled
CCX RM Mode CONFIG_AUTO
CCX RM Config
rm state 0
rm meas interval 0
rts enabled 0
rts threshold 2347
LOMM optimization 0
LOMM Number of Channels 0
channel[0] = 0, channel[1] = 0, channel[2] = 0, channel[3] = 0,
FMC HS AP Type is 0, Threshhold 0
Slot 1
adminstate ADMIN_ENABLED(1)
radioType RADIO_TYPE_80211a
CleanAirAdminState Enabled
countryCode US
countryISOCode US
--More-- chanAutoCfg CONFIG_AUTO
channel 44
channel width 20
extension channel none
txPowerAutoCfg CONFIG_AUTO
txPowerLevel 1
diversitySelection DIVERSITY_ENABLED
htRxAntennaSelection 0F
htTxAntennaSelection 0F
beamformCfg 03
Antenna Mode ANTENNA_OMNI
antennaSelection_0 INTERNAL_ANTENNA
antennaSelection_1 INTERNAL_ANTENNA
antennaSelection_2 INTERNAL_ANTENNA
antennaSelection_3 INTERNAL_ANTENNA
twiceExtAntennaGain 0
Profile Mode CONFIG_AUTO
Load Profile
rfBusyThreshold 0
numClientsThreshold 0
bytesPerSecThreshold 0
Interference Profile
InterferenceThreshold 0
--More-- Noise Profile
NoiseThreshold 0
Coverage Profile
SNRThreshold 0
ExceptionThreshold 0
minClientsThreshold 0
11gSupport Disabled
override mode Disabled
CCX RM Mode CONFIG_AUTO
CCX RM Config
rm state 0
rm meas interval 0
rts enabled 0
rts threshold 2347
LOMM optimization 0
LOMM Number of Channels 0
channel[0] = 0, channel[1] = 0, channel[2] = 0, channel[3] = 0,
FMC HS AP Type is 0, Threshhold 0
AP failure counters: LinkFailure = 277, SpamReboots = 4, ApCrashes = 0
AP join priority = 1
--More-- AP lsc enable = 0
AP lsc reboot cnt = 0
AP lsc max num of Retry = 0
Mesh AP lsc enable = 0
AP retransmit count = 255
AP retransmit timer = 255
AP vlan tag status = Disabled
SSC Controller Hash validation enabled.
12-15-2013 01:46 PM
Get this AP back to your facility where the WLC is located. I want to determine whether or not you've got routing issue or MESH IOS is incorrectly loaded into your AP.
Put the AP in the same subnet as your WLC and see if the AP joins.
12-16-2013 05:39 AM
Ok, umm... there is no way for me to currently move the AP to our Virtual CoLo which is not at the main office. It's in a highly secure data center and I would need to schedule time in advance and not even sure I could hook up the AP inside of the facility. However, I do have a number of this model AP still in the box and so I went ahead and deployed one this morning at our Corporate Facility which is connected accross the WAN like all of our other sites to the Colo. The AP has been up for over 50 minutes and has not disassociated with the controller even once. The few of this same model I have deployed at our other sites have only stayed connected for a couple of minutes at a time.
I have an idea of why this AP may be working here and not at our other locations. Our Corporate office is the only site where the business WLANs are VLANed off from the wired network the guest is of course VLANed everywhere. Now before you start chastising me I have plans to VLAN off the Business wireless at all our sites from the Wired network but I wanted to update all of the APs first namely to address some other wireless issues. So at all of our other sites except Corporate the Business Wireless shares from the same pool as the Business Wired which is VLAN1 by default. So when I created my interfaces on the WLC I did so on the corresponding wired networks. This is the only difference I know of between our Corporate Office and the other sites.
This was not readily apparant to me as a problem at first because the existing 1130AG & 1240AG APs didn't appear to have this issue. I do have most of the APs updated now and was planning on setting up the additoinal networks & VLANs but wasn't sure if this could indeed be causing the issue.
Just remember I inhereted most of this stuff. Luckily I have my CCNA - Routing & Switching Cert now and have some idea of how these should have been properly setup.
Your thoughts?
12-16-2013 01:52 PM
Now before you start chastising me I have plans to VLAN off the Business wireless at all our sites from the Wired network but I wanted to update all of the APs first namely to address some other wireless issues. So at all of our other sites except Corporate the Business Wireless shares from the same pool as the Business Wired which is VLAN1 by default.
No plan of doing that.
Two scenarios are spinning right now:
1. The AP in your Co-Loc could be loaded with MESH IOS;
2. You've got a potential routing problem.
12-16-2013 03:53 PM
Hi Gregory,
Your issue is this AP came with mesh image.
If you cannot get this AP registered to your WLC in mesh mode, then try to reset the AP to factory settings & try to upload correct recovery image (ap3g2-rcvk9w8-tar.152-4.JA1.tar or any previous version) using the mode button option describe in below post.
Then your AP shoud be able to register to your 5508 as a local mode AP. Then change it to FlexConnect if that is the requirement.
http://mrncciew.com/2013/12/13/ap-conversion-using-mode-button/
Give it a try & see
HTH
Rasika
**** Pls rate all useful responses ****
12-16-2013 04:14 PM
Thanks for the advice Leo and Rasika. I actually had my first experience today using a TFTP server to downgrade from Lightweight to Autonomous on an old AP with no certificate so this shouldn't be too difficult. I'll be at one of our facilities tomorrow and will have a chance to test this out. I'll let you know what the results are.
12-19-2013 04:20 AM
try reducing mtu on ap 2600.
12-30-2013 05:33 AM
Hi,
I was out of town for the Holidays and enjoyed some much needed time away from technology so sorry for my delay in getting back to you all. Here is what I have done so far. Rasika, I did change the recovery image on one of those APs to ap3g2-rcvk9w8-tar.152-4.JA1.tar at which time it reloaded the recovery image then connected to the controller and updated to the following version ap3g2-rcvk9w8-tar.152-2.JA1.tar. Once the AP rebooted with the new image from the controller I then ran into the same issue. I haven't had an oppurtunity to attempt reducing the MTU's on the device yet but plan to try that next. What should I change the MTU value to?
Thanks,
12-30-2013 08:58 AM
Hi Gregory,
Pls attach the AP console output this time around to see what's happening.
If you are changing the AP MTU you can do it via below command & set it tio 1363 (TCP MSS size), I think you should have AP registered first to do this
WLC>config ap tcp-adjust-mss enable
You can find more detail about this in below post
http://mrncciew.com/2013/04/07/configuring-tcp-mss/
HTH
Rasika
**** Pls rate all useful responses *****
12-30-2013 10:05 AM
I had forgotten to save the putty log file from my console output when I was on site. I will be sure to attach it next time. Also, I believe the APs I already have registered should stay connected long enough for me to issue the above commands. Thanks again for the info.
Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: