cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
199770
Views
60
Helpful
43
Comments
timsmith
Cisco Employee
Cisco Employee

Problem Description:

Due to the certificate expiration, any new Control and Provisioning of Wireless Access Points (CAPWAP) or Light Weight Access Point Protocol (LWAPP) connection will fail to establish. The main feature that is affected will be the Access Point (AP)-to-controller joining. The secondary feature that is affected will be new mobility connections between the controllers.

When an AP attempts to establish a new connection, the AP fails to join. When you configure mobility between controllers, they will fail to establish a connection.

The likelihood that this issue will be encountered is 100% for wireless products in use (to include APs and controllers) that have a Manufacturer Installed Certificate (MIC) that is older than ten years. Self-Signed Certificates (SSCs) that were generated by the Autonomous-to-lightweight Upgrade Tool will expire on January 1, 2020.

The affected products (listed in the Products Affected section) were released prior to the end of CY2005; beginning in March 2015, the products might begin to experience these symptoms.

Some Cisco CAPWAP based wireless solutions are reaching an age of 10 years from the date of manufacture. When this occurs CAPWAP DTLS tunnels will fail to be established because the certificates on CAPWAP based hardware has expired. The certificate installed in the wireless hardware is used to authenticate the devices when joining the network.

This issue is being tracked via Cisco defect ID: CSCuq19142 and via Field Notice 63942.

*Note: The MIC Lifetime has been documented in past via the Wireless LAN Controller (WLC) Design and Features FAQ at http://www.cisco.com/c/en/us/support/docs/wireless/wireless-lan-controller-software/118833-wlc-design-ftrs-faq.html.
 

Problem Symptom:

Wireless Access Points fail to connect to the Wireless LAN Controller. At the time of the join failure, the WLC's msglog may show messages similar to the following:

Jul 10 16:13:52.443 spam_lrad.c:6164 LWAPP-3-PAYLOAD_ERR: Join request does not contain valid certificate in certificate payload - AP 00:11:22:33:44:55

CAPWAP utilizes Datagram Transport Layer Security (DTLS) in order to encrypt communication between the Lightweight AP and the WLC. The MIC or SSC is used in order to authenticate the Lightweight AP to the WLC, and vice versa, during the DTLS session establishment. The CAPWAP/DTLS connection cannot be established after the MIC or SSC validity end date.

 

Affected Products:

Cisco Wireless LAN Controllers - FCS in 2012 or earlier:


Family / SW Type


Last Software Release


FCS

Date


End of Sale Date


Last Date of Support (HW)


End of Sale Notice

2006 Series Wireless LAN Controller

4.2.x

24/Mar/05

02/Apr/07

21/Apr/12

http://www.cisco.com/c/en/us/products/collateral/wireless/2000-series-wireless-lan-controllers/prod_end-of-life_notice0900aecd805d22b0.html


2100 Series Wireless LAN Controller


7.0.x


09/Jan/07


02/May/12


31/May/17


http://www.cisco.com/c/en/us/products/collateral/wireless/2100-series-wireless-lan-controllers/end_of_life_notice_c51-691053.html


4400 Series Wireless LAN Controller


7.0.x


23/Jun/05


13/Jun/11


30/Jun/16


http://www.cisco.com/c/en/us/products/collateral/switches/catalyst-3750-series-switches/end_of_life_notice_c51-634665.html


Cisco Catalyst 3750G Integrated Wireless LAN Controller


7.0.x


14/Mar/07


13/Jun/11


30/Jun/16


http://www.cisco.com/c/en/us/products/collateral/wireless/catalyst-3750-series-integrated-wireless-lan-controllers/end_of_life_notice_c51-634675.html


Cisco Wireless Services Module 1 (WiSM1)


7.0.x


14/Nov/05


23/Apr/12


30/Apr/17


http://www.cisco.com/c/en/us/products/collateral/interfaces-modules/catalyst-6500-series-7600-series-wireless-services-module-wism/end_of_life_notice_c51-691055.html

NM-AIR-WLC6
(Cisco 6-Access-Point Wireless LAN Controller Network Module)

4.2.x

27/Feb/06

18/Feb/08

16/Feb/13

http://www.cisco.com/c/en/us/products/collateral/interfaces-modules/network-modules/prod_end-of-life_notice0900aecd806aeb34.html

NME-AIR-WLCx (Cisco Wireless LAN Controller Module (WLCM))

7.0.x

15/Feb/07

23/Apr/12

30/Apr/17

http://www.cisco.com/c/en/us/products/collateral/interfaces-modules/wireless-lan-controller-module/end_of_life_notice_c51-691054.html

AIR-CT2504

8.5.x

8/Jul/11

18/Apr/18

30/Apr/23

https://www.cisco.com/c/en/us/products/collateral/wireless/2504-wireless-controller/eos-eol-notice-c51-740645.html

AIR-CT5508

8.5.x

6/May/09

4/May/18

31/Jul/23

https://www.cisco.com/c/en/us/products/collateral/wireless/5500-series-wireless-controllers/eos-eol-notice-c51-740221.html

AIR-CT7510

8.5.x

25/Mar/11

10/Apr/17

30/Apr/22

http://www.cisco.com/c/en/us/products/collateral/wireless/flex-7500-series-wireless-controllers/eos-eol-notice-c51-738009.html

AIR-CT8510

8.5.x

30/Aug/12

3/Sep/18

30/Sep/23

https://www.cisco.com/c/en/us/products/collateral/wireless/8500-series-wireless-controllers/eos-eol-notice-c51-740222.html

WS-SVC-WISM2

8.5.x

2/Apr/11

10/Apr/17

30/Apr/22

https://www.cisco.com/c/en/us/products/collateral/interfaces-modules/wireless-services-module-2-wism2/eos-eol-notice-c51-738008.html

 

 

Cisco Aironet Branded Lightweight Access Points - FCS in 2010 or earlier:

Family / SW Type

Last Software Release

FCS Date

End Of Sale Date

Last date of Support (HW)

End of Sale Notice

Cisco AP801 Integrated Access Point

8.0.x

26-Jun-08 (CISCO888W-GN-A-K9)

31/Mar/16 (C887VA-V-W-E-K9)

31/Mar/21

http://www.cisco.com/c/en/us/products/collateral/routers/800-series-routers/eos-eol-notice-c51-735923.html

Cisco Aironet 1000 Series

4.2.x

24/Mar/05

11/Mar/08

10/Mar/13

http://www.cisco.com/c/en/us/products/collateral/wireless/aironet-1000-series/prod_end-of-life_notice0900aecd806c0c29.html

Cisco Aironet 1040 Series

8.3.x

24/Aug/10

1/Oct/13

30/Sep/18

http://www.cisco.com/c/en/us/products/collateral/wireless/aironet-1140-series/end_of_life_notice_c51-727650.html

Cisco Aironet 1120 Series

7.0.x

02/Oct/02*

19/Jun/09

18/Jun/14

http://www.cisco.com/c/en/us/products/collateral/wireless/aironet-1100-series/eol_c51-506612.html

Cisco Aironet 1130 Series

8.0.x

24/Nov/04*

26/Jul/13

31/Jul/18

http://www.cisco.com/c/en/us/products/collateral/wireless/aironet-1130-ag-series/end_of_life_notice_c51-726426.html

Cisco Aironet 1140 Series

8.3.x

30/Sep/09

1/Oct/13

30/Sep/18

http://www.cisco.com/c/en/us/products/collateral/wireless/aironet-1130-ag-series/end_of_life_notice_c51-727649.html

Cisco Aironet 1200/1230 Series

7.0.x

23/Aug/02*

19/Jun/09

30/Jun/14

http://www.cisco.com/c/en/us/products/collateral/wireless/aironet-1230-ag-series/eol_c51-506614.html

Cisco Aironet 1240 Series

8.0.x

12/Dec/05

26/Jul/13

31/Jul/18

http://www.cisco.com/c/en/us/products/collateral/collaboration-endpoints/unified-ip-phone-7900-series/end_of_life_notice_c51-726425.html

Cisco Aironet 1250 Series

8.0.x

02/Nov/07

20/Jan/12

31/Jan/17

http://www.cisco.com/c/en/us/products/collateral/wireless/aironet-1130-ag-series/end_of_life_notice_c51-681596.html

Cisco Aironet 1260 Series

8.3.x

27/Apr/10

7/Oct/13

2/Jan/18

http://www.cisco.com/c/en/us/products/collateral/wireless/aironet-1260-series/end_of_life_notice_c51-727739.html

Cisco Aironet 1300 Series

7.0.x

04/May/04*

11/Jan/13

31/Jan/18

http://www.cisco.com/c/en/us/products/collateral/wireless/aironet-1300-series/end_of_life_notice_c51-711894.html

Cisco Aironet 1600 Series

8.5.x

16/Nov/12

29/Dec/16

31/Dec/21

https://www.cisco.com/c/en/us/products/collateral/wireless/aironet-1600-series/eos-eol-notice-c51-737506.html 

Cisco Aironet 1700 Series

8.10.x

01/Jun/14

30/Apr/19

30/Apr/24

https://www.cisco.com/c/en/us/products/collateral/wireless/eos-eol-notice-c51-740712.html

Cisco Aironet 2600 Series

8.5.x

18/Jun/12

29/Dec/16

31/Dec/21

https://www.cisco.com/c/en/us/products/collateral/wireless/aironet-2600-series/eos-eol-notice-c51-737512.html

Cisco Aironet 2700 Series

8.10.x

21/Mar/14

30/Apr/19

 30/Apr/24  https://www.cisco.com/c/en/us/products/collateral/wireless/aironet-2700-series-access-point/eos-eol-notice-c51-740711.html

Cisco Aironet 3500 series

8.5.x

26/May/10

1/Apr/16

31/Mar/21

http://www.cisco.com/c/en/us/products/collateral/wireless/aironet-3500-series/eos-eol-notice-c51-734304.html

Cisco Aironet 3600 series

8.5.x

20/Oct/11

29/Dec/16

31/Dec/21

https://www.cisco.com/c/en/us/products/collateral/wireless/aironet-3600-series/eos-eol-notice-c51-737511.html

Cisco Aironet 3700 Series

8.10.x

02/Sep/13

30/Apr/19

30/Apr/24

https://www.cisco.com/c/en/us/products/collateral/wireless/aironet-3700-series/eos-eol-notice-c51-740710.html 

Cisco Aironet AIR-CAP1552I series

8.5.x

2/May/11

30/Mar/16

31/Mar/21

https://www.cisco.com/c/en/us/products/collateral/wireless/aironet-1550-series/eos-eol-notice-c51-735905.html

Cisco Aironet 1570 series: AIR-AP1572EAC and AIR-AP1572EC

-

1/Sep/14

13/Nov/20

30/Nov/25

https://www.cisco.com/c/en/us/products/collateral/wireless/aironet-1570-series/eos-eol-notice-c51-743780.html

Cisco Aironet 1570 series: AIR-AP1572EC3, AIR-AP1572EC4, AIR-AP1572IC3 and AIR-AP1572IC4

-

1/Sep/14

30/Apr/19

30/Apr/24

https://www.cisco.com/c/en/us/products/collateral/wireless/aironet-1570-series/eos-eol-notice-c51-741566.html


*Note: For AP series whose FCS date is before 2005: APs started being manufactured with MICs on July 18, 2005.  Any Lightweight AP's that were manufactured prior to that date have SSCs.

 

Workaround prior to the fix being available:

If you believe you will be affected by this issue and need a fix before the official code with the associated correction is posted at www.cisco.com, then please contact TAC, who will work to provide an escalation release of code accordingly.

 

Recovery for APs in a failed scenario:

*Note: This workaround should only be used in order to allow APs with expired certificates to join the WLC for long enough to upgrade the software.

If the certificates have expired, disable NTP, then change the WLC clock time to a recent earlier time when the certificates were still valid. If you set the clock back too far, newer APs may not be able to join. Once the software has been upgraded, and the affected APs have joined, the WLC clock should be reset to the valid time.

*Note: Temporarily disabling NTP and changing the WLC's time settings can adversely affect other time dependent WLC features such as MFP, SNMPv3, and location.

 

Solution:

To allow additional usage of hardware, beyond the 10 year certificate date, Cisco is providing a software maintenance release with a feature to ignore the validity period of the certificates in the CAPWAP authentication process.

Maintenance releases with the feature to ignore the validity period of the certificates are being created for AireOS 7.0, 7.4 and 8.0.

Cisco has released the fix to Cisco.com in AireOS 7.0.252.0 and 7.4.140.0

Cisco will release to Cisco.com a rebuild of AireOS 8.0 (as version 8.0.120.0) before July 2015.

*Note: Cisco has a beta version of AirOS 8.0 MR2 that does contain the needed commands to work around this issue and can be used until the official AireOS 8.0 MR2 (8.0.120.0) is released on Cisco.com, see the following URL for details:

https://supportforums.cisco.com/document/12492986/80mr2-beta-availability

 

These maintenance releases should be updated before the certificate expires on the APs and WLCs.
 

By default, if an AP and/or WLC certificate has expired, then the DTLS connection will fail. To allow AP's to join a WLC after certificate expiration, upgrade to the fixed software version, then use the following commands:

For 7.0.252.0:
(WLC)>config ap lifetime-check {mic|ssc} enable

For 7.4.140.0 and later:
(WLC)>config ap cert-expiry-ignore {mic|ssc} enable

With "config ap lifetime-check {mic|ssc} enable" or "config ap cert-expiry-ignore {mic|ssc} enable" in effect, the WLC and AP will ignore the expiration date on the devices' MICs and SSCs. The above-noted commands must remain in effect as long as devices with expired MIC or SSC certificates are used.

Because 4400 series WLCs that were among the first manufactured had both Airespace and Cisco MICs installed, with the Airespace MIC being given precedence by the WLC, and the fix for CSCuq19142 is only applicable for Cisco MICs,the currently available fix for CSCuq19142 may not work.  This is potentially applicable to most 4400s manufactured in 2005, and other variants, depending on RMA and refurbishment history of the affected unit.  Please see section "How to Identify Hardware Levels" for how to determine the date of manufacture. If the affected unit was refurbished, the SN may have changed with the MIC remaining the same. At present the only remedy is to disable NTP, then change the WLC clock time to a recent earlier time when the certificates were still valid. Contact TAC to get an escalation image with the fix, as per bug ID CSCuu02970.

 

How to Identify Certificate Expiration date:

(via CLI or Serial number or Python Script or WLCCA)

This section describes how to determine when your AP and WLC MICs and/or SSCs expire using show commands when available or via the device serial number.

1) Manufacturing Installed Certificates (MICs):

The serial number can be used to determine the approximate date when the MIC will expire.

The AP's MIC will expire, at the earliest, ten years past the date of manufacture. Please note, some APs may have more recently created MICs under some conditions. For example, if the AP's motherboard was manufactured and stored, but not assembled until some time later or if the AP was subject to RMA and a refurbishing process, etc.

To determine when the AP was manufactured, run this command on the WLC to find the AP SN:

(Cisco Controller) >show ap inventory all
Inventory for lap1130-sw3-9
NAME: "Cisco AP" , DESCR: "Cisco Wireless Access Point"
PID: AIR-LAP1131AG-E-K9, VID: V01, SN: FCZ1128Q0PE
NAME: "Dot11Radio0" , DESCR: "802.11G Radio"
PID: UNKNOWN, VID: , SN: GAM112706LC
NAME: "Dot11Radio1" , DESCR: "802.11A Radio"
PID: UNKNOWN, VID: , SN: ALP112706LC
The AP chassis SN is in the first section of the output, for example: PID:
AIR-LAP1131AG-E-K9, VID: V01, SN: FCZ1128Q0PE

See "Deriving manufactured date from serial number" section below.

Alternatively, the exact date the MIC expires can be found by running this command and looking for the "Certificate" entry; ignore "CA Certificate" entries. The "end date" associated with the "Validity Date" section is the expiration date for the MIC certificate:

AP_CLI#sh crypto pki certificates
CA Certificate
Status: Available...
...
Certificate
Status: Available
Certificate Serial Number: 728AF4350000001E4C89
Certificate Usage: General Purpose
Issuer:
cn=Cisco Manufacturing CA
o=Cisco Systems
Subject:
Name: C1130-001c58b5b3a4
ea=support@cisco.com
cn=C1130-001c58b5b3a4
o=Cisco Systems
l=San Jose
st=California
c=US
CRL Distribution Points:
http://www.cisco.com/security/crl/cmca.crl
Validity Date:
start date: 04:22:10 UTC Jul 11 2007
end date: 04:32:10 UTC Jul 11 2017
Associated Trustpoints: Cisco_IOS_MIC_cert


2) Self-Signed Certificates (SSCs):

In order to determine if you have an SSC, run this WLC command:

AP_CLI >show auth-list
...
AP with Self-Signed Certificate................ yes
...

All AP SSC's have an expiration date of January 1st, 2020.


3) Wireless LAN Controllers (WLCs):

You can determine the WLC's serial number by running this command:
WLC_CLI>show inventory

Burned-in MAC Address............................ 24:E9:B3:43:C4:E0
Maximum number of APs supported.................. 75
NAME: "Chassis" , DESCR: "Cisco 2500 Series Wireless LAN Controller"
PID: AIR-CT2504-K9, VID: V04, SN: PSZ17441ANT

To determine the WLC serial number via the GUI, navigate: Controller > Inventory

If you have AireOS 8.0 or later, to determine when the WLC certificate expires, run this command and look for the "Cisco SHA1 device cert":

WLC_CLI: show certificate all

Certificate Name: Cisco SHA1 device cert
Subject Name :
C=US, ST=California, L=San Jose, O=Cisco Systems, CN=AIR-CT2504-K9-d0c282d65a20, MAILTO=support@cisco.com
Issuer Name :
O=Cisco Systems, CN=Cisco Manufacturing CA
Serial Number :
454384735992863371807890
Validity :
Start : 2011 Jul 26th, 20:17:17 GMT
End : 2021 Jul 26th, 20:27:17 GMT
Signature Algorithm :
rsa-pkcs1-sha1
Hash key :
SHA1 Fingerprint : 98:89:eb:12:2a:98:bc:fe:ad:5b:8f:23:63:0f:47:d1:36:ce:f5:be
MD5 Fingerprint : ba:f3:98:9a:cd:f8:01:08:84:b8:66:3c:6a:6c:d3:05

This command is not available in AireOS releases prior to 8.0. There is no similarly straightforward command to derive this date in earlier AireOS releases. As an alternate method, use the WLC serial numbers to determine the earliest possible MIC expiration date.

Deriving manufactured date from product serial numbers:
The serial number format is: "LLLYYWWSSSS"; where "YY" is the year of manufacture and "WW" is the week of manufacture. The date code can be found in the 4 middle digits of the serial number.

Manufacturing Year Codes:
01 = 1997 06 = 2002 11 = 2007 16 = 2012
02 = 1998 07 = 2003 12 = 2008 17 = 2013
03 = 1999 08 = 2004 13 = 2009 18 = 2014
04 = 2000 09 = 2005 14 = 2010
05 = 2001 10 = 2006 15 = 2011


Manufacturing Week Codes:
01-05 = January,   15-18 = April,      28-31 = July,              41-44 = October
06-09 = February,  19-22 = May,      32-35 = August,         45-48 = November
10-14 = March,      23-27 = June,     36-40 = September,   49-52 = December

Example: SN FCZ1128Q0PE has year code 11, meaning it was manufactured in the year 2007. The week code is 28, meaning it was manufactured in July of that year.

 

4) Access Point Certificate Check Tool:

A Python script has been written that runs on Windows, Mac and Linux systems that allows a user to check on the certificate expiration date for all AP's on their network. 

The following Cisco Support Forum's article explains how to access and run this tool:

            Access Point Certificate Check Tool - apCertCheck

 

5) Wireless LAN Controller (WLC) Config Analyzer:

WLCCA version 3.6.5 and above has support to check the AP Certificate expiration date.  This check is done based on the AP Serial number and will flag any AP needing checked based on if the AP serial number is within 60 days of expiration. 

The following Cisco Support Forum's article explains how to access and run this tool:

          WLC Config Analyzer

NOTE: Using the AP Serial number is only an approximation of the MIC expiration date.  Any AP's flagged by this method should always be check for the real MIC expiration date via the Access Point commands listed.

 

Comments
Vinay Sharma
Level 7
Level 7

Thank you Tim for sharing  the key information with CSC users.

 

Regards,

Vinay Sharma

Community Manager

CCIE#44972

bechong
Cisco Employee
Cisco Employee

Hi Tim,

There's a typo:

"The AP lifetime-check parameter is disabled by default. After upgrading to the fixed software, disable the AP lifetime-check using this command:"

Instead it should read:

"The AP lifetime-check parameter is disabled by default. After upgrading to the fixed software, enable the AP lifetime-check using this command:"

Thought I'd point this out since the CLI command is confusing/counter intuitive enough.

timsmith
Cisco Employee
Cisco Employee

Actually according to development, the wording is correct.  Basically if you want the controller to ignore the AP Certificate lifetime check, you have to "enable" the command, if the command is disabled (which is the default), then the AP Certificate is checked.   We have a bug filed to get this command syntax cleaned up since its very confusing based on what the command reads.

Jerry Cao
Level 1
Level 1

Does certificate expiration effect autonomous APs in any way?

timsmith
Cisco Employee
Cisco Employee

Autonomous mode AP does not use the MIC in this way, so it would not be affected.

pinglis
Level 7
Level 7

Can someone clarify if the listed access points models are only affected when connecting to a one of the listed controllers (e.g. 4400) or if they will also have issues connecting to a 5500 series controller (e.g. running 7.6.130.0)

timsmith
Cisco Employee
Cisco Employee

There are MICs installed on both controllers and AP's, if you have an AP with an expired MIC, it would not be able to join a 5508 even if the 5508 MIC is still valid. 

Using the updated code and appropriate command on the controller, as noted in the solution section above, the AP's with expired MICs will still be able to join.

pinglis
Level 7
Level 7

Thanks for the quick response. Do you know if there will be a fix for 7.6 and if not why not?

My slight issue is 8.0 is not compatible with the version of Prime we are currently running.

timsmith
Cisco Employee
Cisco Employee

There will be no Cisco.com release for 7.6 code with the fix as there are no more scheduled maintenance releases for 7.6, but if 7.6 is needed we do have an escalation image that will have the Cert fix in it.  Just open a TAC case and you can get the code.

 

nick-davies
Level 1
Level 1

Has anyone succeeded in getting version 7.0.252.0?

I've tried but it requires a service contract, but when I try and get a service contract it tells me the units are out of support and can't have a service contract allocated to them...

So I'm stuck with the prospect of scrapping the unit.  Cisco.com support have been totally intransigent and frankly not helpful in suggesting any way out of this Catch-22.

Thanks

timsmith
Cisco Employee
Cisco Employee

Nick,

  Send me an email: timsmith@cisco.com, let me see what I can do to help.

 

-Tim

Leo Laohoo
Hall of Fame
Hall of Fame
I've tried but it requires a service contract, but when I try and get a service contract it tells me the units are out of support and can't have a service contract allocated to them...

I agree with Tim.  Send an email to TAC and they will publish the firmware fix for you.

 

Also, read THIS.  This promo ends at the end of July 2015.  Good time (and price) for anyone to get an upgrade of the WLC when you get a free WLC 2504 (with 25 AP license) for free.

Shawn Purdy
Level 1
Level 1

Can someone confirm the syntax of the command for checking the AP;s?  When I do a "show AP inventory all"  I get Cisco AP name is invalid.  The syntax I am told to use is the following. 

show ap inventory <Cisco AP>

not sure what name to use.  I've the AP model and the actual DNS name of the AP no luck. Anyone else run into this?

Aaron
Cisco Employee
Cisco Employee

Shawn, what AireOS version are you running?  "show ap inventory all" may not be in your version.

In any case, you can use "show ap summary" to see the AP names (as known by the WLC) that are joined.  Here's an example:

 

(WLC3750) >show ap inventory Deb1250

NAME: "Cisco AP"    , DESCR: "Cisco Wireless Access Point"
PID: AIR-LAP1252AG-A-K9   ,  VID: V01,  SN: FTX114690WB


NAME: "Dot11Radio0"    , DESCR: "802.11N 2.4GHz Radio"
PID: AIR-RM1252G-A-K9,  VID: V01,  SN: FOC11325N3Q


NAME: "Dot11Radio1"    , DESCR: "802.11N 5GHz Radio"
PID: AIR-RM1252A-A-K9,  VID: V01,  SN: FOC11411HVE

 

Shawn Purdy
Level 1
Level 1

THanks that did it.  I'm using 7.4.110.  I was typing in the name wrong.  I used an AP with a shorter name and that worked. 

 

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: