vWLC 7.4 and AP 1602 - CAPWAP fails

Answered Question
Jul 1st, 2013
User Badges:

Hi guys!


In my lab, everything just worked fine. Now AP1602 is on customer site. AP gets vWLC IP address via DHCP option 43, 60. If I try to debug vWLC console with this command "debug capwap detail enable":


(Cisco Controller) >debug capwap detail enable

*spamApTask0: Jul 01 12:04:26.669: 68:86:a7:cb:f6:d0 CAPWAP Control Msg Received from 10.10.10.215:16281


*spamApTask0: Jul 01 12:04:26.683: 68:86:a7:cb:f6:d0 CAPWAP Control Msg Received from 10.10.10.215:16281


*spamApTask0: Jul 01 12:04:26.690: 68:86:a7:cb:f6:d0 CAPWAP Control Msg Received from 10.10.10.215:16281


*spamApTask0: Jul 01 12:04:26.690: 68:86:a7:cb:f6:d0 DTLS connection 0x10fb84e0 closed by controller

*spamApTask0: Jul 01 12:04:26.691: 68:86:a7:cb:f6:d0 CAPWAP Control Msg Received from 10.10.10.215:16281


*spamApTask0: Jul 01 12:04:26.691: CAPWAP DTLS connection closed msg


*spamApTask2: Jul 01 12:05:09.168: 00:1f:6c:8a:4d:41 CAPWAP Control Msg Received from 10.10.10.156:57832


*spamApTask2: Jul 01 12:05:09.168: 34:a8:4e:ba:47:40 packet received of length 123 from 10.10.10.156:57832


*spamApTask2: Jul 01 12:05:09.168: 34:a8:4e:ba:47:40 Msg Type = 1 Capwap state = 0


*spamApTask2: Jul 01 12:05:09.168: 34:a8:4e:ba:47:40 msgEleLength = 1 msgEleType = 20


*spamApTask2: Jul 01 12:05:09.168: 34:a8:4e:ba:47:40 Total msgEleLen = 94


*spamApTask2: Jul 01 12:05:09.168: 34:a8:4e:ba:47:40 msgEleLength = 40 msgEleType = 39


*spamApTask2: Jul 01 12:05:09.168: 34:a8:4e:ba:47:40 Total msgEleLen = 50


*spamApTask2: Jul 01 12:05:09.168: 34:a8:4e:ba:47:40 msgEleLength = 1 msgEleType = 41


*spamApTask2: Jul 01 12:05:09.168: 34:a8:4e:ba:47:40 Total msgEleLen = 45


*spamApTask2: Jul 01 12:05:09.168: 34:a8:4e:ba:47:40 msgEleLength = 1 msgEleType = 44


*spamApTask2: Jul 01 12:05:09.168: 34:a8:4e:ba:47:40 Total msgEleLen = 40


*spamApTask2: Jul 01 12:05:09.168: 34:a8:4e:ba:47:40 msgEleLength = 10 msgEleType = 37


*spamApTask2: Jul 01 12:05:09.168: 34:a8:4e:ba:47:40 Vendor specific payload from AP  34:A8:4E:BA:47:40 validated


*spamApTask2: Jul 01 12:05:09.168: 34:a8:4e:ba:47:40 Total msgEleLen = 26


*spamApTask2: Jul 01 12:05:09.168: 34:a8:4e:ba:47:40 msgEleLength = 22 msgEleType = 37


*spamApTask2: Jul 01 12:05:09.168: 34:a8:4e:ba:47:40 Vendor specific payload from AP  34:A8:4E:BA:47:40 validated


*spamApTask2: Jul 01 12:05:09.168: 34:a8:4e:ba:47:40 Total msgEleLen = 0


*spamApTask2: Jul 01 12:05:09.168: 34:a8:4e:ba:47:40 1. 0 0


*spamApTask2: Jul 01 12:05:09.168: 34:a8:4e:ba:47:40 2. 232 3


*spamApTask2: Jul 01 12:05:09.168: 34:a8:4e:ba:47:40 3. 0 0


*spamApTask2: Jul 01 12:05:09.168: 34:a8:4e:ba:47:40 4. 200 0


*spamApTask2: Jul 01 12:05:09.168: 34:a8:4e:ba:47:40 Discovery resp: AC Descriptor message element len = 40


*spamApTask2: Jul 01 12:05:09.168: 34:a8:4e:ba:47:40 acName = Cisco_92:e4:7b


*spamApTask2: Jul 01 12:05:09.168: 34:a8:4e:ba:47:40 Discovery resp:AC Name message element length = 58


*spamApTask2: Jul 01 12:05:09.168: 34:a8:4e:ba:47:40 Discovery resp: WTP Radio Information msg length = 67


*spamApTask2: Jul 01 12:05:09.168: 34:a8:4e:ba:47:40 Discovery resp: CAPWAP Control IPV4 Address len = 77


*spamApTask2: Jul 01 12:05:09.168: 34:a8:4e:ba:47:40 Discovery resp: CAPWAP Control IPV6 Address len = 99


*spamApTask2: Jul 01 12:05:09.168: 34:a8:4e:ba:47:40 Discovery resp: Mwar type payload len = 110


*spamApTask2: Jul 01 12:05:09.168: 34:a8:4e:ba:47:40 Discovery resp: Time sync payload len = 125


*spamApTask2: Jul 01 12:05:09.168: 34:a8:4e:ba:47:40 WTP already released




On Web interface Management->Logs->Message logs-> "DTLS-3-HANDSHAKE_FAILURE: openssl_dtls.c:681 Failed to complete DTLS handshake with peer 10.10.10.156


Do you have any ideas , why it doesn't work? Why DTLS connection is closed by vWLC?

Correct Answer by Scott Fella about 4 years 1 month ago

Well the workaround for that was the newer rcv image.  With the new rcv image, you didn't have to do that.  Since the 1602's come with that newer version, you shouldn't of had that issue.


Thanks,

Scott

Help out other by using the rating system and marking answered questions as "Answered"

  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 5 (1 ratings)
Loading.
EvaldasOu Mon, 07/01/2013 - 02:40
User Badges:

While debugging vWLC:


*spamApTask2: Jul 01 12:17:50.173: 34:a8:4e:ba:47:40 DTLS connection found! Acquiring lock for 0x10fba700

*spamApTask2: Jul 01 12:17:50.173: 34:a8:4e:ba:47:40 Called... for connection 0x10fba700

*spamApTask2: Jul 01 12:17:50.173: 34:a8:4e:ba:47:40 record=Alert epoch=0 seq=2

*spamApTask2: Jul 01 12:17:50.173: openssl_shim_info_callback: SSL state = 0x2180; where = 0x4004; ret = 0x22a

*spamApTask2: Jul 01 12:17:50.173: openssl_shim_info_callback: ret_type_string=fatal

*spamApTask2: Jul 01 12:17:50.173: openssl_shim_info_callback: ret_desc_string=bad certificate

*spamApTask2: Jul 01 12:17:50.173: openssl_shim_info_callback: SSL_state_string=SSLv3 read client certificate A

*spamApTask2: Jul 01 12:17:50.173: openssl_shim_info_callback: SSL state = 0x2180; where = 0x2002; ret = 0x0

*spamApTask2: Jul 01 12:17:50.173: openssl_shim_info_callback: ret_type_string=unknown

*spamApTask2: Jul 01 12:17:50.173: openssl_shim_info_callback: ret_desc_string=close notify

*spamApTask2: Jul 01 12:17:50.173: openssl_shim_info_callback: SSL_state_string=SSLv3 read client certificate A

*spamApTask2: Jul 01 12:17:50.173: 34:a8:4e:ba:47:40 SSL_do_handshake: SSL_ERROR_SSL while communicating with 10.10.10.156 : sslv3 alert bad certificate

*spamApTask2: Jul 01 12:17:50.173: 34:a8:4e:ba:47:40  Requested by openssl_dtls_process_packet

*spamApTask2: Jul 01 12:17:50.173: 34:a8:4e:ba:47:40          in openssl_dtls.c:687

*spamApTask2: Jul 01 12:17:50.173: dtls_conn_hash_delete: Deleting hash for Local 10.10.11.243:5246  Peer 10.10.10.156:57832


*spamApTask2: Jul 01 12:17:50.173: 34:a8:4e:ba:47:40 Called...

*spamApTask2: Jul 01 12:17:50.173: 34:a8:4e:ba:47:40 Shutdown completed


Is it possible that there is wrong time settings applied on the AP? Can I send NTP settings for AP with DHCP option 42, from windows server?

Leo Laohoo Mon, 07/01/2013 - 03:53
User Badges:
  • Super Gold, 25000 points or more
  • Hall of Fame,

    The Hall of Fame designation is a lifetime achievement award based on significant overall achievements in the community. 

  • Cisco Designated VIP,

    2017 LAN, Wireless

Post the following output:


1.  vWLC:  sh sysinfo;

2.  vWLC:  sh time;

3.  AP:  sh version;

4.  AP:  sh inventory

Shaoqin Li Mon, 07/01/2013 - 06:57
User Badges:
  • Bronze, 100 points or more

can you make the AP first finish join a physical wlc then move it to the vWLC?

Sent from Cisco Technical Support iPad App

EvaldasOu Mon, 07/01/2013 - 13:16
User Badges:

Hi guys,


I was on site. I connected console cable directly to the AP , and I used these commands:

#test capwap erase

#test capwap restart


After that, I pulled off Ethernet cable (AP was powered with PoE). Then I plugged in PoE, AP loaded... And everything was working like a charm...


I think that the problem occurred because before real deployment, I connected my AP to the vWLC in my lab... I just wanted to test it... I left the AP in FlexConnect mode, and sent it to the customer.


Could anyone comment "test capwap erase", "test capwap restart" commands? These commands are "hidden" in CLI, but it's also very helpful commands... As you can see in my posts above, AP and vWLC DTLS connection failed before...

Scott Fella Mon, 07/01/2013 - 14:47
User Badges:
  • Super Silver, 17500 points or more
  • Hall of Fame,

    The Hall of Fame designation is a lifetime achievement award based on significant overall achievements in the community. 

  • Cisco Designated VIP,

    2017 Wireless

I typically use that command when I have AP's that have issues connecting. The erase I have only had to use a few times but test capwap controller IP or even the capwap ap controller IP address has worked well for me. If all fails, I erase the nvram and any files or images and put the rcv in the AP. that's usually fixes everything.

Sent from Cisco Technical Support iPhone App

EvaldasOu Tue, 07/02/2013 - 01:05
User Badges:

Guys,


I think I am talking about this bug here : CSCua55382 . We can find more details here :

http://www.cisco.com/image/gif/paws/113677/virtual-wlan-dg-00.pdf


Known Issue: AP(s) not joining vWLC − The AP must get the hash entry from a legacy controller before it

joins a vWLC.


• An AP must be at software version 7.3.1.35 and above to successfully join a virtual controller. Virtual

controllers use SSC in order to validate an AP before joining.

•An AP at version 7.3 can validate the SSC certificate provided by the virtual controller.

• After successful certificate validation, an AP will check the hash key of the virtual controller in the

list of stored keys in flash. If it matches the stored hash, validation is passed and the AP moves to the

RUN state. If hash validation fails, it will disconnect from the controller and restart the discovery

process.

• The hash validation, which is an extra authorization step, will be performed only if the AP is joining a

virtual controller. There will be a knob to turn on/off hash key validation.

• By default, hash validation is enabled, which means that the AP needs to have the virtual controller

hash key in its flash before it can successfully complete association with the virtual controller. If the

knob is turned off, the AP will bypass the hash validation and move directly to the RUN state.

• The hash key can be configured in the controller mobility configurations, which gets pushed to all the

APs which are joined. The AP will save this configuration until it successfully associates to another

controller. After which, it inherits the hash key configuration from the new controller.

• Typically, APs can join a traditional controller, download the hash keys, and then join a virtual

controller. However, if it is joined to a traditional controller, the hash validation knob can be turned

off and it can join any virtual controller. The administrator can decide to keep the knob on or off

This information is captured in Cisco bug ID CSCua55382.


Exceptions:

•If the AP does not have any hash key in its flash, it will bypass the hash validation, assuming that it is

a first time installation.

     ♦In this case, the hash validation is bypassed irrespective of whether the hash validation knob

is on/off.

     ♦ Once it successfully joins the controller, it will inherit the mobility group member hash configuration (if configured in the controller). After which, it can join a virtual controller only if it has a hash key entry in its database.

• Clearing the AP configuration from the controller or on the AP console will result in the erasing of all

the hash keys. After which, the AP joins the virtual controller as if it is a first time installation.


♦AP> test capwap erase

♦AP> test capwap restart


So... because I connected my AP to the vWLC in my lab, it downloaded hash keys.Without erasing these keys, AP was unable to establish DTLS tunnel with another vWLC.


Hope that helps!

Correct Answer
Scott Fella Tue, 07/02/2013 - 04:45
User Badges:
  • Super Silver, 17500 points or more
  • Hall of Fame,

    The Hall of Fame designation is a lifetime achievement award based on significant overall achievements in the community. 

  • Cisco Designated VIP,

    2017 Wireless

Well the workaround for that was the newer rcv image.  With the new rcv image, you didn't have to do that.  Since the 1602's come with that newer version, you shouldn't of had that issue.


Thanks,

Scott

Help out other by using the rating system and marking answered questions as "Answered"

Actions

This Discussion