Cisco Support Community
cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
Community Member

vWLC 7.6.120 with cap2602 APs

Running virtual wireless controller 7.6.120 with AIR-CAP2602E-K-K9 model access points. The server crashed a few days ago and now the access points arent able to join to the controller. They join to the controller for about 5 minutes, and i see users actually connecting to it. Then it just disappears. and does the same cycle again. I've also "clear all config" on all of the access points as well. This setup is actually on a drillship and it goes over the VSAT, we have them setup locally onboard at other sites and never had issues until now. I've attached the logs on its behavior on what it does.

I've also upgraded the controller from 7.4.100 to 7.6.120 thinking it would fix the issue. No dice.

Any help would be appreciated.

Everyone's tags (1)
13 REPLIES
Hall of Fame Super Blue

Error message looks like the

Error message looks like the APs are running MESH IOS.

Community Member

Mesh IOS? Could you explain?

Mesh IOS? Could you explain?

Community Member

I downgraded the controller.

I downgraded the controller. See the following below. 

 

(Cisco Controller) >show sysinfo

Manufacturer's Name.............................. Cisco Systems Inc.
Product Name..................................... Cisco Controller
Product Version.................................. 7.4.121.0
RTOS Version..................................... 7.4.121.0
Bootloader Version............................... 7.4.121.0
Emergency Image Version.......................... 7.4.121.0

Build Type....................................... DATA + WPS

System Name...................................... RIG201-vWLC
System Location.................................. 
System Contact................................... 
System ObjectID.................................. 1.3.6.1.4.1.9.1.1631
IP Address....................................... 10.254.201.224
System Up Time................................... 0 days 4 hrs 39 mins 57 secs
System Timezone Location......................... (GMT -6:00) Central Time (US and Canada)
System Stats Realtime Interval................... 5
System Stats Normal Interval..................... 180

Configured Country............................... Multiple Countries:KE,US


--More-- or (q)uit
State of 802.11b Network......................... Enabled
State of 802.11a Network......................... Enabled
Number of WLANs.................................. 5
Number of Active Clients......................... 0

Memory Current Usage............................. Unknown
Memory Average Usage............................. Unknown
CPU Current Usage................................ Unknown
CPU Average Usage................................ Unknown

Burned-in MAC Address............................ 00:0C:29:03:42:BC
Maximum number of APs supported.................. 200

--------------------------------------------------------------------------------

AP4c00.82b9.96de#sh inventory 
NAME: "AP2600", DESCR: "Cisco Aironet 2600 Series (IEEE 802.11n) Access Point"
PID: AIR-CAP2602E-K-K9 , VID: V01, SN: FGL1729W06B

--------------------------------------------------------------------------------

AP4c00.82b9.96de#sh version 
Cisco IOS Software, C2600 Software (AP3G2-K9W8-M), Version 15.2(2)JB3, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2013 by Cisco Systems, Inc.
Compiled Thu 19-Dec-13 04:30 by prod_rel_team

ROM: Bootstrap program is C2600 boot loader
BOOTLDR: C2600 Boot Loader (AP3G2-BOOT-M) LoaderVersion 12.4(25e)JA1, RELEASE SOFTWARE (fc1)

AP4c00.82b9.96de uptime is 3 minutes
System returned to ROM by power-on
System image file is "flash:/ap3g2-k9w8-mx.152-2.JB3/ap3g2-k9w8-xx.152-2.JB3"
Last reload reason: 

 

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

cisco AIR-CAP2602E-K-K9    (PowerPC) processor (revision A0) with 180214K/81920K bytes of memory.
Processor board ID FGL1729W06B
PowerPC CPU at 800Mhz, revision number 0x2151
Last reset from power-on
LWAPP image version 7.4.121.0
1 Gigabit Ethernet interface
2 802.11 Radios

32K bytes of flash-simulated non-volatile configuration memory.
Base ethernet MAC Address: 4C:00:82:B9:96:DE
Part Number                          : 73-14511-02
PCA Assembly Number                  : 800-37898-01
PCA Revision Number                  : A0
PCB Serial Number                    : FOC17273MC9
Top Assembly Part Number             : 800-38357-01
Top Assembly Serial Number           : FGL1729W06B
Top Revision Number                  : A0
Product/Model Number                 : AIR-CAP2602E-K-K9   

 

Configuration register is 0xF

 

 

Hall of Fame Super Blue

Configured Country...........

Configured Country............................... Multiple Countries:KE,US

This is the country codes configured for your WLC.  You've got Kenya and US.  Enable both is not a good idea because both wireless communication regulatory domain are very dfferent. 

PID: AIR-CAP2602E-K-K9 

This is your AP.  The "-K" means your AP is manufactured for the regulatory domain of South Korea.  

 

If you want this AP to join the WLC, disable Kenya and US country code in the WLC and enable South Korea country code and then reboot the WLC.

Community Member

Leo,Thanks for the prompt

Leo,

Thanks for the prompt reply.

KE is Korea Extended

KN is Kenya

Hall of Fame Super Blue

KE is Korea Extended

KE is Korea Extended
KN is Kenya

In your book, KE can be called "Korea Extended".  

 

In internation rules, KE stands for Kenya.  Cisco follows the international rules.  

 

For more information, go HERE.

 

WTF.  Disregard previous comment.  I haven't seen a country code of South Korea as "KE" before.  Mea culpa.

Community Member

So i'll just select kenya

So i'll just select kenya below and it should register right? Here's the screen shot below.

 

Hall of Fame Super Blue

Enable "KE" and reboot

Enable "KE", disable "US" and reboot controller.  

 

NOTE:  I've never seen South Korea as a "KE" country code before.  My deepest and sincerest apologies.  

Community Member

No worries. I've always had

No worries. I've always had it as KE previously when it worked.

I believe i know why they aren't connecting. When i upgraded from 7.4.100 to the latest 7.6.120. It showed that all of the APs got the image, but when i tried to reboot them it would keep saying, "AP is still being upgraded". It lasted about 3 hours, so i decided to save the configs, then reset the virtual machine, which then caused some of the APs to have a corrupt image. 80% of the images went into Rommon Mode, so i had to restore it via TFTP.

I went ahead and rebuilt the controller from scratch on the 7.4.121 IOS, and installed an IOS image on the APs. Hopefully this fix the issues from the APs to join to the controller.

One thing that i noticed is that the APs will all be up, and when I have another one plugged in, all of them would disappear. Could you explain the reasoning for that?

Hall of Fame Super Blue

One thing that i noticed is

One thing that i noticed is that the APs will all be up, and when I have another one plugged in, all of them would disappear. Could you explain the reasoning for that?

I don't have a vWLC nor have I seen a strange behaviour like this before.

 It lasted about 3 hours, so i decided to save the configs, then reset the virtual machine, which then caused some of the APs to have a corrupt image.

The APs are capable of holding two IOS.  So let's presume you rebooted the WLC, if the primary IOS got corrupted, the APs should've booted the second one.  

 

Besides, I've seen this kind of behaviour several times and I would force the WLC to reboot.  This is a known WLC firmware bug that started appearing in 7.4.X.X.

 

Make sure your vWLC is running the latest 7.4.X.X firmware.

Community Member

yeah its still doing it again

yeah its still doing it again. it stays up for about 5 mins. then it disassociates itself from the controller.

 

*apfReceiveTask: Aug 10 21:30:43.977: dc:a5:f4:1c:42:a0 Received LWAPP Down event for AP dc:a5:f4:1c:42:a0 slot 0!
*apfReceiveTask: Aug 10 21:30:43.977: dc:a5:f4:1c:42:a0 Received LWAPP Down event for AP dc:a5:f4:1c:42:a0 slot 1!
*spamApTask7: Aug 10 21:30:43.977: dc:a5:f4:1c:42:a0 DTLS connection closed event receivedserver (10:254:201:224/5246) client (10:254:201:115/48708)
*spamApTask7: Aug 10 21:30:43.977: dc:a5:f4:1c:42:a0 Entry exists for AP (10:254:201:115/48708)
*spamApTask7: Aug 10 21:30:43.977: dc:a5:f4:1c:42:a0 apfSpamProcessStateChangeInSpamContext: Deregister LWAPP event for AP dc:a5:f4:1c:42:a0 slot 0
*spamApTask7: Aug 10 21:30:43.977: dc:a5:f4:1c:42:a0 apfSpamProcessStateChangeInSpamContext: Deregister LWAPP event for AP dc:a5:f4:1c:42:a0 slot 1
*spamApTask7: Aug 10 21:30:43.977: dc:a5:f4:1c:42:a0 No AP entry exist in temporary database for 10.254.201.115:48708 
*spamApTask6: Aug 10 21:30:43.977: dc:a5:f4:1c:42:a0 Discovery Request from 10.254.201.115:48707

*spamApTask6: Aug 10 21:30:43.977: dc:a5:f4:1c:42:a0 Join Priority Processing status = 0, Incoming Ap's Priority 1, MaxLrads = 200, joined Aps =5
*spamApTask6: Aug 10 21:30:43.977: dc:a5:f4:1c:42:a0 Discovery Response sent to 10.254.201.115:48707

*spamApTask6: Aug 10 21:30:43.977: dc:a5:f4:1c:42:a0 Discovery Response sent to 10.254.201.115:48707

*apfReceiveTask: Aug 10 21:30:43.977: dc:a5:f4:1c:42:a0 Deregister LWAPP event for AP dc:a5:f4:1c:42:a0 slot 0
*apfReceiveTask: Aug 10 21:30:43.977: dc:a5:f4:1c:42:a0 Deregister LWAPP event for AP dc:a5:f4:1c:42:a0 slot 1
*spamApTask0: Aug 10 21:30:44.052: dc:a5:f4:1c:3f:10 Msg Timeout for 10.254.201.137:48703, max retries: 5

*spamApTask7: Aug 10 21:30:46.190: f8:4f:57:f0:b3:80 Discovery Request from 10.254.201.133:26122

*spamApTask7: Aug 10 21:30:46.190: f8:4f:57:f0:b3:80 apfSpamProcessStateChangeInSpamContext: Down LWAPP event for AP f8:4f:57:f0:b3:80 slot 0
*spamApTask7: Aug 10 21:30:46.191: f8:4f:57:f0:b3:80 apfSpamProcessStateChangeInSpamContext: Down LWAPP event for AP f8:4f:57:f0:b3:80 slot 1
*spamApTask7: Aug 10 21:30:46.191: f8:4f:57:f0:b3:80 Finding DTLS connection to delete for AP (10:254:201:133/26121)
*spamApTask7: Aug 10 21:30:46.191: f8:4f:57:f0:b3:80 Disconnecting DTLS Capwap-Ctrl session 0x8f90240 for AP (10:254:201:133/26121)

*spamApTask7: Aug 10 21:30:46.191: f8:4f:57:f0:b3:80 CAPWAP State: Dtls tear down

*spamApTask7: Aug 10 21:30:46.192: f8:4f:57:f0:b3:80 Discovery Response sent to 10.254.201.133:26122

*spamApTask7: Aug 10 21:30:46.192: f8:4f:57:f0:b3:80 Discovery Response sent to 10.254.201.133:26122

Community Member

i actually got a chance to

i actually got a chance to get into one of the APs. Here's what it does.

 

RIG201-NAVDECK-ELECT#
*Aug 10 16:13:13.999: %DTLS-5-SEND_ALERT: Send FATAL : Close notify Alert to 10.254.201.224:5246
*Aug 10 16:13:24.059: %CAPWAP-3-ERRORLOG: Go join a capwap controller 
*Aug 10 16:14:24.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: 10.254.201.224 peer_port: 5246
*Aug 10 16:14:53.999: DTLS_CLIENT_ERROR: ../capwap/base_capwap/dtls/base_capwap_dtls_connection_db.c:2176 Max retransmission count reached!
*Aug 10 16:15:23.999: %DTLS-5-SEND_ALERT: Send FATAL : Close notify Alert to 10.254.201.224:5246
*Aug 10 16:15:23.999: %CAPWAP-3-ERRORLOG: Go join a capwap controller 
*Aug 10 16:14:24.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: 10.254.201.224 peer_port: 5246
*Aug 10 16:14:53.999: DTLS_CLIENT_ERROR: ../capwap/base_capwap/dtls/base_capwap_dtls_connection_db.c:2176 Max retransmission count reached!
*Aug 10 16:15:23.999: %DTLS-5-SEND_ALERT: Send FATAL : Close notify Alert to 10.254.201.224:5246
*Aug 10 16:15:34.059: %CAPWAP-3-ERRORLOG: Go join a capwap controller 
*Aug 10 16:16:34.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: 10.254.201.224 peer_port: 5246

*Aug 10 16:17:03.999: DTLS_CLIENT_ERROR: ../capwap/base_capwap/dtls/base_capwap_dtls_connection_db.c:2176 Max retransmission count reached!
*Aug 10 16:17:33.999: %DTLS-5-SEND_ALERT: Send FATAL : Close notify Alert to 10.254.201.224:5246
*Aug 10 16:17:33.999: %CAPWAP-3-ERRORLOG: Go join a capwap controller 
*Aug 10 16:16:34.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: 10.254.201.224 peer_port: 5246
*Aug 10 16:16:34.215: %CAPWAP-5-DTLSREQSUCC: DTLS connection created sucessfully peer_ip: 10.254.201.224 peer_port: 5246
*Aug 10 16:16:34.215: %CAPWAP-5-SENDJOIN: sending Join Request to 10.254.201.224
*Aug 10 16:16:34.219: %CAPWAP-3-ERRORLOG: Invalid event 10 & state 5 combination.
*Aug 10 16:16:34.219: %CAPWAP-3-ERRORLOG: CAPWAP SM handler: Failed to process message type 10 state 5.
*Aug 10 16:16:34.219: %CAPWAP-3-ERRORLOG: Failed to handle capwap control message from controller
*Aug 10 16:16:34.219: %CAPWAP-3-ERRORLOG: Failed to process encrypted capwap packet from 10.254.201.224
*Aug 10 16:16:34.323: Starting Ethernet promiscuous mode
*Aug 10 16:16:35.931: %LWAPP-4-CLIENTEVENTLOG: OfficeExtend Localssid saved in AP flash
*Aug 10 16:16:36.335: %CAPWAP-5-JOINEDCONTROLLER: AP has joined controller RIG201-vWLC
*Aug 10 16:16:36.399: %LWAPP-4-CLIENTEVENTLOG: No Flex ACL map configuration file to load. Connect to controller to get configuration file
*Aug 10 16:16:36.399: %LWAPP-4-CLIENTEVENTLOG: No Flex ACL map configuration file to load. Connect to controller to get configuration file
*Aug 10 16:16:36.399: %LWAPP-4-CLIENTEVENTLOG: No LS Flex ACL map configuration file to load. Connect to controller to get configuration file
*Aug 10 16:16:36.399: %LWAPP-4-CLIENTEVENTLOG: No Central Dhcp map configuration file to load. Connect to controller to get configuration fileWLAN id 5, SSID W-RMSCorp, L2ACL , L2ACL AP 

*Aug 10 16:16:36.399: %LWAPP-3-CLIENTERRORLOG: Switching to Connected mode
*Aug 10 16:16:45.447: %WIDS-6-ENABLED: IDS Signature is loaded and enabled
*Aug 10 16:17:01.671: %CLEANAIR-6-STATE: Slot 0 disabled
*Aug 10 16:17:01.671: %CLEANAIR-6-STATE: Slot 1 disabled


., 4)10 16:23:03.999: %CAPWAP-3-ERRORLOG: Retransmission count for packet exceeded max(CAPWAP_WTP_EVENT_REQUEST
*Aug 10 16:23:04.003: %LWAPP-3-CLIENTERRORLOG: Switching to Standalone mode
*Aug 10 16:23:04.007: %CAPWAP-3-ERRORLOG: GOING BACK TO DISCOVER MODE
*Aug 10 16:23:04.007: %DTLS-5-SEND_ALERT: Send FATAL : Close notify Alert to 10.254.201.224:5246
*Aug 10 16:23:04.067: %WIDS-6-DISABLED: IDS Signature is removed and disabled.
*Aug 10 16:23:14.071: %CAPWAP-3-ERRORLOG: Go join a capwap controller 
RIG201-NAVDECK-ELECT#
*Aug 10 16:23:04.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: 10.254.201.224 peer_port: 5246
*Aug 10 16:23:33.999: DTLS_CLIENT_ERROR: ../capwap/base_capwap/dtls/base_capwap_dtls_connection_db.c:2176 Max retransmission count reached!
*Aug 10 16:24:03.999: %DTLS-5-SEND_ALERT: Send FATAL : Close notify Alert to 10.254.201.224:5246
*Aug 10 16:24:03.999: %CAPWAP-3-ERRORLOG: Go join a capwap controller 
*Aug 10 16:23:04.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: 10.254.201.224 peer_port: 5246

Cisco Employee

The LAP, if doesn't receive

The LAP, if doesn't receive any packet in past 30 seconds, then it retransmits the packet
for 5 times after that, at an interval of 3 seconds. So no communication to wlc on control
channel for 45 seconds will result in tearing down of tunnel. WLC on the other hand takes
90 seconds to determine that ap had lost communication.

In your logs, the WTP event message was delivered to wlc and had a payload in it, which
tried to inform wlc about a link failure. The error is pretty generic in nature, and there
could have been many problems that AP could have deduced as link failure.

For example, in one of the cases, it was dynamic arp inspection

We could not narrow down this, only after we went to the switch and looked at the
following logs:-


May 29 14:12:03: %PM-4-ERR_DISABLE: arp-inspection error detected on Gi1/0/44, putting

Gi1/0/44 in err-disable state

May 29 14:12:04: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/44,

changed state to down

May 29 14:12:05: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/44, changed state to down

May 29 14:17:03: %PM-4-ERR_RECOVER: Attempting to recover from arp-inspection err-disable

state on Gi1/0/44

May 29 14:17:23: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/44, changed state to up

May 29 14:17:25: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/44,

changed state to up


After applying ip arp inspection trust for all the APs, the service was back.



So we will have to go in that direction. Please check in your network, what all occurred
in that duration.
441
Views
0
Helpful
13
Replies
CreatePlease to create content