just noticed this one the other day, neither of our mobility engines are syncing with the WLCs.
Our setup is:
all WLCs are 22.214.171.124
MSE3355 - was 126.96.36.199
vMSE - was 188.8.131.52
all was fine. We upgraded both MSEs v184.108.40.206, due to the deferral notice for 220.127.116.11. Ever since, nothing will sync. All we get is NMSP status is inactive for both MSEs to all WLCs. The message is hashkey mismatch between MSE and WLC. I have tried numerous things to no avail, like deleting the key from the WLCs. When I try to re-sync, I do see the key gets pushed from the MSE to the WLC. But in the MSE logs I do see certificate unknown errors when trying to sync. I do have a case open on this.
It really seems like it resulted from the MSE upgrade, but I didn't see anything in the release notes that caused any concern.
Has anyone seen this symptom at all? Would it really be as simple as finding and deleting the key store? Any comments are appreciated.
Thanks - chris
am also seeing on the WLC, via command line, the following when doing a sho nmsp statis summ:
SSL Handshake failed............................. 15542
and it is incrementing. So it definitely looks like either:
/opt/mse/locsrv/ssl or /var/mse/certs/nss
has something corrupt. But based on a different post, I don't think it's the nss directory, as apache is running fine, so I think it is the /opt/mse/locsrc/ssl directory. But I don't want to do anything that will make things any worse.
Did you ever get this resolved?
I'm having a similiar issue. I have a wlc 8500 running v8.0.1, MSE is v8 and Prime is V2.1.
When i add mse on prime, MSE is added to the Auth list on the WLC automatically as a LBS-SSC. IF i change that to a SSC MSE complain about a hash key mismatch.
When i click NMSP status i get a time mismatch. i have set all 3 servers to sync to the same ntp server. The wlc is set to GMT, Prime and MSE are set to BST. When i click the NMSP status it says there is a time issue, but it shows that the wlc time and the MSE time are exactly the same.
Not sure what else to try apart from a MSE rebuild.
You need to define the new hashkey. Here is a support link that guides you to obtaining that hashkey:
HI scott, Thanks for replying.
The hash key seems to be already defined as per below, once i added MSE it seemed to auto generate with the WLC. i have attached the screenshot error on nmsp status.
I can also see the ssl Handshake errors increasing
(Cisco Controller) >show auth-list
Authorize MIC APs against Auth-list or AAA ...... disabled
Authorize LSC APs against Auth-List ............. disabled
APs Allowed to Join
AP with Manufacturing Installed Certificate.... yes
AP with Self-Signed Certificate................ no
AP with Locally Significant Certificate........ no
Mac Addr Cert Type Key Hash
----------------------- ---------- ------------------------------------------
00:0c:29:de:aa:4c LBS-SSC 62d6c2d230f87615b1583394277f4cf59a451d96
cmd> show server-auth-info
invoke command: com.aes.server.cli.CmdGetServerAuthInfo
AesLog queue high mark: 50000
AesLog queue low mark: 500
Server Auth Info
MAC Address: 00:0c:29:de:aa:4c
SHA1 Key Hash: 62d6c2d230f87615b1583394277f4cf59a451d96
SHA2 Key Hash: 8579084679da0a14b0b07c3ca6b262d12b0a0a4ea3521668e1922d62f42ad1f6
Certificate Type: SSC
yeah have removed and re added a few times. have rebooted the MSE a few times. I haven't rebooted the WLC as yet but might try that later tonight when there are no clients connected.
Just to let you all know, i got it resolved. it seems MSE version 8 uses SHA 256. So i copied that string from MSE and changed to SHA 256 on the controller and it worked straight away,
That was also the fix for me. In the WLC GUI delete the MSE created MAC address under Security>AAA>AP Polices. You can not create an AP Authorization with SHA2(256) with the GUI. Go to the cmd line of the WLC and run.
config auth-list add sha256-lbs-ssc (MAC of the MSE in xx:yy format) SHA2 Key Hash
There are some differences in your setup from mine, but I definitely had a hash mismatch and a manual copy fixed things. As for the time sync thing, I really don't know if the different timezones would cause this or not. But it seems from the screenshot that it's not even trying to establish the nmsp connection due to this. Maybe Scott or someone can chime in here regarding timezones. - chris
Brian, and all,
basically, the result is that you have to manually copy the hash. FIrst, determine the hash on the MSE, and then copy it via CLI onto the controller. I did find a document that was specific to the "converged access" gear, but is applicable to all gear I guess nowadays.
The link to the document is http://www.cisco.com/c/en/us/support/docs/wireless/5700-series-wireless-lan-controllers/117477-technote-addmac-00.html, and should get you going.
Please respond whether or not this worked for you, and I can send you some other stuff from the case offline. Thanks - chris
This discussion was very helpful. We had the same problem too. Cisco has created a Bug ID for tracking the issue: CSCuq50069 - SHA1 key cipher not working between WLC 80 and MSE 80 CCO versions. The Bug appears to be resolved in 18.104.22.168 code, but no workaround was mentioned. I was able to resolve the issue by SSH'ing to each of the MSE 3365's, logging in, issuing the show server-auth-info command and copying the output to a notepad file. Once I had the MSE's mac address and the SHA2 hash, I SSH'ed to the WLC and from the CLI I entered this command: config auth-list add sha256-lbs-ssc <Mac address > <40bit Key> and replaced the Mac address for that of the MSE and the 40bit Key with that of the SHA2 hash. This resolved my issue.