Solved: 1231G LWAPP 12.3(7)JX rebooting over and over

AnthonyRowe · ‎06-09-2006

I recently converted a 1231G from 12.3(7)JA IOS to LWAPP 12.3(7)JX. The AP now tries to join the controller (4404)but receives no join response and reboots. I have heard a rumor that there may be a problem with the certificate that's created on the AP during the conversion for certain older 1231's but can't find anything about it on Cisco's site. Does anyone know where I can find documentation on how to fix this? The errors from the AP log follow:

*Mar 1 00:00:23.473: %LWAPP-5-CHANGED: LWAPP changed state to DISCOVERY

AP0011.5c40.6f8d>

Translating "CISCO-LWAPP-CONTROLLER.vassar.edu"...domain server (143.229.1.3)

*Mar 1 00:00:32.247: %DHCP-6-ADDRESS_ASSIGN: Interface FastEthernet0 assigned D

HCP address 172.29.100.179, mask 255.255.248.0, hostname AP0011.5c40.6f8d

*Mar 1 00:00:33.249: LWAPP_CLIENT_ERROR: lwapp_name_lookup - Could Not resolve

CISCO-LWAPP-CONTROLLER.vassar.edu

*Mar 1 00:00:44.200: %LWAPP-5-CHANGED: LWAPP changed state to JOIN

*Mar 1 00:00:50.201: LWAPP_CLIENT_ERROR_DEBUG: spamHandleJoinTimer: Did not rec

ieve the Join response

*Mar 1 00:00:50.201: LWAPP_CLIENT_ERROR_DEBUG: No more AP manager IP addresses

remain.

*Mar 1 00:00:50.201: %SYS-5-RELOAD: Reload requested by LWAPP CLIENT. Reload Re

ason: DID NOT GET JOIN RESPONSE.

*Mar 1 00:00:50.201: %LWAPP-5-CHANGED: LWAPP changed state to DOWNXmodem file s

ystem is available.

Stephen Rodriguez · ‎06-20-2006

the time can be an issue. If you are using NTP, let them sit overnight to get up to date, and in synch with the controller usually works.

Remember, that when you run the conversion tool, you can specify the time from the AP or the pc running the tool. So if you're pc is also synched to an NTP, then use the pc time when you do the conversion.

One quick way to see if it is a time issue, is console into the AP, and watch the clock, if it is off, you can change the ntp offset on the controller to see if that helps to get in to join.

HTH,
Steve

------------------------------------------------------------------------------------------------
Please remember to rate useful posts, and mark questions as answered

View solution in original post

Stephen Rodriguez · ‎06-09-2006

It's not a rumor. When you convert an AP from an IOS, to LWAPP, there is a certificate issue. If the AP you converted was manufactured before July 2005, it does not have a Cisco MIC, Manufactured Installed Certificate, instead you will get an SSC, Self Signed Certificate.

If you look in the folder where you installed the Upgrade Tool, you will have a csv file with that SSC hash stored there. Then you can go to the controller and import that into it.

HTH,
Steve

------------------------------------------------------------------------------------------------
Please remember to rate useful posts, and mark questions as answered

AnthonyRowe · ‎06-09-2006

Thank you steprodr. I will check into this. The errors don't specifically mention that a certificate is being rejected so maybe it's something else. I found a similiar question posted back in april where option 43 on the dhcp server wasn't configured with the vendor specific attribs or something for the 1200's? I'm going to be checking that avenue out as well. Thanks for the info!

Stephen Rodriguez · ‎06-09-2006

well, if the access point finds the controller, it doesn't sound like option 43/60 is the problem. If you want to find out if it is a certificate issue you can issue:

debug mac addr

debug lwapp events enable

From there you will see more information as to what is happening when the AP is trying to join the controller.

HTH,
Steve

------------------------------------------------------------------------------------------------
Please remember to rate useful posts, and mark questions as answered

AnthonyRowe · ‎06-09-2006

Ok, that will help alot. thanks again!

mmacmahon · ‎06-12-2006

One other thing, is your AP on a different subnet than the controller? If it is then the DHCP option 43 will resolve the issue. You can make it talk to the controller by entering the following command at the console port "test lwapp controller ip " This will force the AP to look for a controller at that address. If you go to the controller and look under Security>AP Policies, you can see if there is a cert hash added for the AP under AP authorization list. The issue I see repeatedly is that the conversion looks successful, however the tool fails to install the key hash for the AP on the Controller. I have had this issue running the tool on windows 2k prof. No issues on XP.

AnthonyRowe · ‎06-12-2006

The AP and controllers are on different subnets and running in layer3 mode. I am running the tool on XP.

Ok, I'll look into this. Thanks alot!

zhenningx · ‎06-13-2006

My APs upgrade were successfuly, but no APs listed under Security -> AP Policies. Does it mean there is still something wrong with mine? Is AP authorization list required?

Stephen Rodriguez · ‎06-13-2006

Not necessrily, if your AP's were manufactured after Jly 2005, you will have a MIC that is preinstalled, and you won't need a SSC.

HTH,
Steve

------------------------------------------------------------------------------------------------
Please remember to rate useful posts, and mark questions as answered

mmacmahon · ‎06-13-2006

one of two issues

1. AP and controller on different subnets? if yes then you need to config option 43 in the DHCP scope for the APs.

2. If your APs are like mine (manufactured prior to 7-2005) then you could be having the same issue I am with the tool on win2k. Go to the Upgrade tool install folder and look fo a csv file. it should contain the MAC address, SSC, then a hash if there is no hash then your hosed and need to recover it. The csv file is generated by the upgrade tool and you can use it to push out the AP Auth list on multiple controller through the WCS software. See my previous reply for a link to the instructions.

hope this helps, michael

AnthonyRowe · ‎06-16-2006

Ok, I have been trying the suggestions and here's what I've discovered: the AP's are on the same subnet as the controllers. When I add the mac addr, ssc, key hash to WCS under "ap authorization" it gets distributed correctly to all 6 controllers (4404's) properly. The first one I converted is now working however, the rest are not, rebooting over and over. A debug on one of the controllers follows:

(Cisco Controller) >Wed Jun 14 13:35:47 2006: Received LWAPP DISCOVERY REQUEST f

rom AP 00:11:5c:40:6f:98 to ff:ff:ff:ff:ff:ff on port '29'

Wed Jun 14 13:35:47 2006: Successful transmission of LWAPP Discovery-Response to

AP 00:11:5c:40:6f:98 on Port 29

Wed Jun 14 13:35:58 2006: Received LWAPP JOIN REQUEST from AP 00:11:5c:40:6f:98

to 06:0a:10:10:00:00 on port '29'

Wed Jun 14 13:35:58 2006: LWAPP Join-Request does not include valid certificate

in CERTIFICATE_PAYLOAD from AP 00:11:5c:40:6f:98.

Wed Jun 14 13:35:58 2006: Unable to free public key for AP 00:11:5C:40:6F:98

Wed Jun 14 13:35:58 2006: spamDeleteLCB: stats timer not initialized for AP 00:1

1:5c:40:6f:98

Wed Jun 14 13:35:58 2006: spamProcessJoinRequest : spamDecodeJoinReq failed

Wed Jun 14 13:36:10 2006: spamDeleteLCB: stats timer not initialized for AP 00:1

1:92:5e:97:b0

Wed Jun 14 13:36:10 2006: spamProcessJoinRequest : spamDecodeJoinReq failed

Wed Jun 14 13:36:30 2006: spamDeleteLCB: stats timer not initialized for AP 00:1

1:92:5e:98:60

Wed Jun 14 13:36:30 2006: spamProcessJoinRequest : spamDecodeJoinReq failed

Wed Jun 14 13:36:40 2006: Received SPAM_UPLOAD_ROGUE_TABLE_ENTRY

Wed Jun 14 13:37:12 2006: Received LWAPP DISCOVERY REQUEST from AP 00:11:5c:40:6

f:98 to ff:ff:ff:ff:ff:ff on port '29'

Wed Jun 14 13:37:12 2006: Successful transmission of LWAPP Discovery-Response to

AP 00:11:5c:40:6f:98 on Port 29

Wed Jun 14 13:37:23 2006: Received LWAPP JOIN REQUEST from AP 00:11:5c:40:6f:98

to 06:0a:10:10:00:00 on port '29'

Wed Jun 14 13:37:23 2006: LWAPP Join-Request does not include valid certificate

in CERTIFICATE_PAYLOAD from AP 00:11:5c:40:6f:98.

Wed Jun 14 13:37:23 2006: Unable to free public key for AP 00:11:5C:40:6F:98

Wed Jun 14 13:37:23 2006: spamDeleteLCB: stats timer not initialized for AP 00:1

1:5c:40:6f:98

Wed Jun 14 13:37:23 2006: spamProcessJoinRequest : spamDecodeJoinReq failed

Wed Jun 14 13:37:52 2006: spamDeleteLCB: stats timer not initialized for AP 00:1

1:92:5e:97:b0

Wed Jun 14 13:37:52 2006: spamProcessJoinRequest : spamDecodeJoinReq failed

Wed Jun 14 13:38:14 2006: spamDeleteLCB: stats timer not initialized for AP 00:1

1:92:5e:98:60

Wed Jun 14 13:38:14 2006: spamProcessJoinRequest : spamDecodeJoinReq failed

Wed Jun 14 13:38:37 2006: Received LWAPP DISCOVERY REQUEST from AP 00:11:5c:40:6

f:98 to ff:ff:ff:ff:ff:ff on port '29'

Wed Jun 14 13:38:37 2006: Successful transmission of LWAPP Discovery-Response to

AP 00:11:5c:40:6f:98 on Port 29

Wed Jun 14 13:38:40 2006: Received SPAM_UPLOAD_ROGUE_TABLE_ENTRY

Wed Jun 14 13:38:48 2006: Received LWAPP JOIN REQUEST from AP 00:11:5c:40:6f:98

to 06:0a:10:10:00:00 on port '29'

Wed Jun 14 13:38:48 2006: LWAPP Join-Request does not include valid certificate

in CERTIFICATE_PAYLOAD from AP 00:11:5c:40:6f:98.

Wed Jun 14 13:38:48 2006: Unable to free public key for AP 00:11:5C:40:6F:98

Wed Jun 14 13:38:48 2006: spamDeleteLCB: stats timer not initialized for AP 00:1

1:5c:40:6f:98

Wed Jun 14 13:38:48 2006: spamProcessJoinRequest : spamDecodeJoinReq failed

ED CARMODY · ‎06-17-2006

Anthony, a couple things:

First, you should have opened a TAC case to get this resolved. I think you are experiencing a known isssue.

Second, it appears to me that the certs that the tool wrote to the APs are invalid. I have seen this exact situation before, where either a bug in the controller code, or a time discrepancy between the controlller, APs and/or laptop running the upgrade utility causes the cert to be invalid - "LWAPP Join-Request does not include valid certificate"...likely the dates on the certs are wrong, exceeding the validity interval.

I think the bad news is, you need to convert the APs back to IOS and re-run the conversion tool...but you have to be local to the AP to hold down the reset button while it boots. How many APs do you need to do this to?

AnthonyRowe · ‎06-20-2006

I was wondering if the date and time could be off too much on something. I'm looking into that today. If not I'm looking at a bug then?

I'm currently concentrating on one building with 4 AP's in it but once I work out the kinks I have something like 65 more to do in several other buildings.

Thanks for the advice! I'll let you know what I find today while I'm on-site.

Stephen Rodriguez · ‎06-20-2006

the time can be an issue. If you are using NTP, let them sit overnight to get up to date, and in synch with the controller usually works.

Remember, that when you run the conversion tool, you can specify the time from the AP or the pc running the tool. So if you're pc is also synched to an NTP, then use the pc time when you do the conversion.

One quick way to see if it is a time issue, is console into the AP, and watch the clock, if it is off, you can change the ntp offset on the controller to see if that helps to get in to join.

HTH,
Steve

------------------------------------------------------------------------------------------------
Please remember to rate useful posts, and mark questions as answered

rfuchser · ‎06-21-2006

Hi

use this doucment to install and configure SSC on the controller. Hope this help.

Rolf