02-17-2012 03:48 PM
Hello,
I am supporting an environment that has 30+ remote WAEs deployed with a CM at the HQ.
All remote WAE's Versions = Cisco Wide Area Application Services (universal-k9) Software Release 4.2.3b (build b4 Oct 4 2010)
HQ's CM version = Cisco Wide Area Application Services (universal-k9) Software Release 4.4.3 (build b4 Aug 22 2011)
On 4 of these WAEs, I currently am receiving encryption key alarms:
WAE#show alarms detail support
Critical Alarms:
----------------
Alarm ID Module/Submodule Instance
--------------- -------------------- ---------------
1 mstore_key_retrieval cms ssl_mstore_key
Apr 11 18:36:16.026 CDT, Processing Error Alarm, #000002, 3000:700008
Unable to generate and/or retrieve SSL managed store encryption key from the Key Manager
/alm/crit/cms/mstore_key_retrieval_failure:
CMS/Management agent failed to generate and/or retrieve SSL managed store encryption key from Key Manager.
Explanation:
This alarm indicates one of following issues: Central
Manager device(s) is not reachable. Secure store on
Central Manager is initialized but not open. Key Manager
process on Central Manager device is not running or failing
to respond. Key Manager is unable to process key
generation or retrieval request. If this issue is
present, the WAE device will not be able to process a
configuration update received from the Central Manager if
it contains SSL certificate/key information.
Action:
Check if Central Manager device is reachable (TCP
connections from the WAE to the Central Manager on port
443) Check following log files for additional information
about the error: /local1/errorlog/kc.log on WAE
/local1/errorlog/km/km.log on CM
2 mstore_key_failure sslao mstore_key_failure
Apr 11 18:39:07.518 CDT, Processing Error Alarm, #000006, 26000:26002
Failed to open SSL store due to failure in getting key from Central Manager.
/alm/crit/sslao/mstore_key_failure:
SSL managed secure store key retrieval failure.
Explanation:
The SSL accelerator is unable to get the SSL secure store
key from the Central Manager.
Action:
Check the connection with the Central Manager.
The explanations and actions match the alarm book , but in addition to that, in the Cisco WAAS Monitoring Guide, it also states:
Alarm 700008 (mstore_key_retrieval_failure) CMS/Management agent failed to generate and/or retrieve SSL managed store encryption key from Key Manager.
Severity: Critical
Category: Processing
Description: This alarm indicates one of following issues:
–The WAAS Central Manager device is not reachable
–Secure store on WAAS Central Manager is initialized but not open
–The Key Manager process on the WAAS Central Manager device is not running or failing to respond
–Key Manager cannot process key generation or retrieval request. If this issue is present, the WAAS device cannot process a configuration update received from WAAS Central Manager if it contains SSL certificate and key pair information.
Action: Check to see if the WAAS Central Manager device is reachable (TCP connections from the WAE to the WAAS Central Manager on port 443). Check the following log files for additional information about the error:
–On WAE: /local1/errorlog/kc.log on WAE
–On WAAS Central Manager: /local1/errorlog/km/km.log
Action: Fix the clock on the device or the primary WAAS Central Manager.
For a complete list of alarm conditions, see the Alarm Book located in the WAAS 4.2.1 Software Download area on Cisco.com.
Using this information, I've checked the following:
TCP 443 is reachable from the WAE to the CM (I can telnet from each WAE to the CM on TCP 443)
Time is correct on the WAEs and CM ('show ntp status' and 'show clock' are consistent)
Secure store on CM is open ('show cms secure-store' on the CM shows that the mode is in 'Open' state),
Verified that the key manager process is running (Looking at the CM's KM log shows plenty of action that it's working for other WAEs)
Here is some information I gathered from the WAEs' kc.log files and the CM's km.log (slightly scrubbed):
From the WAEs' kc.log files:
pool-1-thread-1] INFO CommClientAbstractRPC - Send key retrieval request to CM 10.x.x.x for token d1b77e45-ce60-4332-a92d-3d3cb17d35cf
pool-1-thread-1] WARN CommClientAbstractRPC - Received error response from KM(20,No key found for token d1b77e45-ce60-4332-a92d-3d3cb17d35cf from device 17111)
From the CM's km.log file:
[pool-1-thread-4] INFO - retrieveKey request, token=d1b77e45-ce60-4332-a92d-3d3cb17d35cf from device WAE1/17111
[pool-1-thread-4] INFO - Checking secure store open
[pool-1-thread-4] INFO - Loading KEK from data server
[pool-1-thread-4] INFO - ticket 17111 (1327767406332, 1327767392433, 13899, 10000)
[pool-1-thread-4] WARN - No key found for token d1b77e45-ce60-4332-a92d-3d3cb17d35cf from device 17111
*** Going through these logs, I've seen other devices have the same issue, and eventually a WAE records the following:
[main] ERROR DeviceInfo - /state/node.dat (No such file or directory)
java.io.FileNotFoundException: /state/node.dat (No such file or directory)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(Unknown Source)
at java.io.FileInputStream.<init>(Unknown Source)
at com.cisco.waas.kc.DeviceInfo.retrieveNodeInfo(DeviceInfo.java:65)
at com.cisco.waas.kc.DeviceInfo.<init>(DeviceInfo.java:47)
at com.cisco.waas.kc.DeviceInfo.getInstance(DeviceInfo.java:37)
at com.cisco.waas.kc.comm.CommClientAbstractRPC.retrieveKey(CommClientAbstractRPC.java:149)
at com.cisco.waas.kc.RetrieveKeyCommand.execute(RetrieveKeyCommand.java:43)
at com.cisco.waas.cli.CLICommand.execute(CLICommand.java:114)
at com.cisco.waas.cli.AbstractCLI.process(AbstractCLI.java:28)
at com.cisco.waas.kc.KeyClient.main(KeyClient.java:40)
[main] ERROR DeviceInfo - /state/node.dat (No such file or directory)
java.io.FileNotFoundException: /state/node.dat (No such file or directory)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(Unknown Source)
at java.io.FileInputStream.<init>(Unknown Source)
at com.cisco.waas.kc.DeviceInfo.retrieveNodeInfo(DeviceInfo.java:65)
at com.cisco.waas.kc.DeviceInfo.<init>(DeviceInfo.java:47)
at com.cisco.waas.kc.DeviceInfo.getInstance(DeviceInfo.java:37)
at com.cisco.waas.kc.comm.CommClientAbstractRPC.initKey(CommClientAbstractRPC.java:40)
at com.cisco.waas.kc.InitKeyCommand.execute(InitKeyCommand.java:40)
at com.cisco.waas.cli.CLICommand.execute(CLICommand.java:114)
at com.cisco.waas.cli.AbstractCLI.process(AbstractCLI.java:28)
at com.cisco.waas.kc.KeyClient.main(KeyClient.java:40)
*** Followed with what appears to be a new SSL key being generated ***:
[main] INFO DeviceInfo - loaded device info, hash H04Fer5il3b/9oanDZXx/7aBnIo=
[pool-1-thread-1] DEBUG CMProber$ProbeWorker - Sending CM probe request to CM 10.x.x.x
[pool-1-thread-1] DEBUG CMProber$ProbeWorker - CM 10.x.x.x returned :primary:4.4.3.0.4
[pool-1-thread-1] DEBUG CMProber$ProbeWorker - Primary CM address 10.x.x.x version 4.4.3.0.4
[main] DEBUG CommClientAbstractRPC - CM version 4.4.3
[main] INFO CommClientAbstractRPC - Send key initialization request to CM 10.x.x.x key type SSL
[main] INFO CommClientAbstractRPC - Received new token for generated key SSL/cbe3d6fc-875e-4b61-baeb-528c55cb3597
[main] INFO DeviceInfo - loaded device info, hash H04Fer5il3b/9oanDZXx/7aBnIo=
[pool-1-thread-1] INFO CommClientAbstractRPC - Send key retrieval request to CM 10.0.65.234 for token cbe3d6fc-875e-4b61-baeb-528c55cb3597
[main] INFO CommClientAbstractRPC$1 - Successfully retrieved key from CM for token cbe3d6fc-875e-4b61-baeb-528c55cb3597
*** And the CM records the following ***:
[pool-1-thread-4] INFO - initKey request from device WAE2/30129 key type SSL
[pool-1-thread-4] INFO - Checking secure store open
[pool-1-thread-4] INFO - Loading KEK from data server
[pool-1-thread-4] INFO - Return crypto of type : 0
[pool-1-thread-4] INFO - Checking secure store open
[pool-1-thread-4] INFO - Loading KEK from data server
[pool-1-thread-4] INFO - Loading KEK from data server
[pool-1-thread-4] INFO - Generated new key WAE2/SSL token cbe3d6fc-875e-4b61-baeb-528c55cb3597
I'm wanting to know why this occurs on some boxes and not others, and what triggers the process for a WAE to stop repeatedly sending key retrieval requests with a token that the CM has repeatedly replies with the key not being found and performing an initial key request.
Thanks!
08-25-2012 04:57 PM
Hello Jazz
I'm not sure if this issue has been resolved. We have encountered the same issue on some of our WAE device, it happened after power hit but i am uncertain if that what triggers the alarm. What we did was deregister/register WAE from CMS and disable/enable accelerator SSL.
Register/deregister WAE to CMS
cms deregister force
cms enable
Disable/enable the ssl service
no accelerator ssl enable
accelerator ssl enable
Hope it helps!
Message was edited by: Rebecca Doctor
01-21-2013 06:22 AM
Hi all, I got into the same issue and looking at a solution I found a way to clear those alarms whithout re-registering the WAE/WAVE. Here it goes...
WAE##sh accelerator
Accelerator Licensed Config State Operational State
----------- -------- ------------ -----------------
cifs Yes Enabled Running
epm Yes Enabled Running
http Yes Enabled Running
mapi Yes Enabled Running
nfs Yes Enabled Running
ssl Yes Enabled Disabled ---> your SSL AO is probably down due the issue
video No Enabled Shutdown
WAE#sh alarms
Critical Alarms:
----------------
Alarm ID Module/Submodule Instance
--------------- -------------------- ---------------
1 mstore_key_retrieval cms ssl_mstore_key
2 mstore_key_failure sslao mstore_key_failure
Major Alarms:
-------------
None
Minor Alarms:
-------------
None
WAE#crypto pki managed-store initialize
All certificate/private keys in SSL managed store will be deleted and optimized SSL traffic will be interrupted. Are you sure you want to continue(yes/no)? [no]:yes
Restarting SSL accelerator. Done.
After a couple of minutes alarms will be cleared and SSLAO will be back UP.
WAE#sh accelerator
Accelerator Licensed Config State Operational State
----------- -------- ------------ -----------------
cifs Yes Enabled Running
epm Yes Enabled Running
http Yes Enabled Running
mapi Yes Enabled Running
nfs Yes Enabled Running
ssl Yes Enabled Running
video No Enabled Shutdown
WAE#sh alarms
Critical Alarms:
----------------
None
Major Alarms:
-------------
None
Minor Alarms:
-------------
None
In case you have the issue in the Core WAE (where the cms secure-store is opened), you might need to initialize it.
Regards,
Fernando
02-13-2017 07:20 AM
Gentleman,
I have the same issue, I was able to get the first alarm to go off by dereg/ register WAE, and I left off with this alarm.
Critical Alarms:
----------------
Alarm ID Module/Submodule Instance
--------------- -------------------- ---------------
1 mstore_key_retrieval cms ssl_mstore_key
Major Alarms:
-------------
None
Minor Alarms:
-------------
None
I am not running any SSL accel so I am not sure why I am getting this alarm any suggestions?
Accelerator Licensed Config State Operational State
----------- -------- ------------ -----------------
cifs Yes Disabled Shutdown
epm Yes Enabled Running
http Yes Enabled Running
mapi Yes Disabled Shutdown
nfs Yes Enabled Running
ssl Yes Disabled Shutdown <SHUT>
video No Disabled Shutdown
wansecure Yes Enabled Initializing
smb Yes Enabled Running
ica Yes Disabled Shutdown
Thanks in Advance,
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide