I'm using WAVE-574-K9 with OS version 4.1.5f and faced "CMS status offline" a few weeks ago.
I recovered to online by "no cms enable" and "cms enable" commands.
I also investigated below.
If my investigation result is correct, I'd like to know root cause of momery error and solution.
Any one have an idea?
[Investigation results]
1. "config-sync" is started every 5 minutes.
2. "Time of last config-sync" was "Sun Apr 1 17:39:11 2012" when I found CMS status offline.
3. I found some log messages which was generated on "2012 Apr 1 17:44:14".
4. As result 1 to 3, the cause of this issue seems memory error.
[The log what I operated]
w001#show alarms detail
Critical Alarms:
----------------
None
Major Alarms:
-------------
None
Minor Alarms:
-------------
None
w001#
w001#
w001#show cms info
Device registration information :
Device Id = 5176
Device registered as = WAAS Application Engine
Current WAAS Central Manager = 10.72.4.33
Registered with WAAS Central Manager = 10.72.4.33
Status = Offline
Time of last config-sync = Sun Apr 1 17:39:11 2012
CMS services information :
Service cms_ce is running
w001#
w001#
w001#
w001#conf t
w001(config)#no cms enable
management services stopped
w001(config)#exit
w001#show cms info
Device registration information :
Device Id = 5176
Device registered as = WAAS Application Engine
Current WAAS Central Manager = 10.72.4.33
Registered with WAAS Central Manager = 10.72.4.33
Status = Offline
Time of last config-sync = Sun Apr 1 17:39:11 2012
CMS services information :
Service cms_ce is not running
w001#conf t
w001(config)#cms enable
Please preserve running configuration using 'copy running-config startup-config'.
Otherwise management service will not be started on reload and node will be shown
'offline' in WAAS Central Manager UI.
management services enabled
w001(config)#
w001(config)#
w001(config)#
w001(config)#
w001(config)#
w001(config)#exit
w001#show cms info
Device registration information :
Device Id = 5176
Device registered as = WAAS Application Engine
Current WAAS Central Manager = 10.72.4.33
Registered with WAAS Central Manager = 10.72.4.33
Status = Offline
Time of last config-sync = Sun Apr 1 17:39:11 2012
CMS services information :
Service cms_ce is running
w001#
w001#show cms info
Device registration information :
Device Id = 5176
Device registered as = WAAS Application Engine
Current WAAS Central Manager = 10.72.4.33
Registered with WAAS Central Manager = 10.72.4.33
Status = Online
Time of last config-sync = Mon Apr 2 11:37:02 2012
CMS services information :
Service cms_ce is running
w001#
w001#show version
Cisco Wide Area Application Services Software (WAAS)
Copyright (c) 1999-2010 by Cisco Systems, Inc.
Cisco Wide Area Application Services Software Release 4.1.5f (build b2 Apr 20 2010)
Version: oe574-4.1.5f.2
Compiled 12:26:34 Apr 20 2010 by cnbuild
System was restarted on Sun Mar 20 04:14:48 2011.
The system has been up for 1 year, 2 weeks, 7 hours, 25 minutes, 13 seconds.
w001#
[/local1/syslog.txt]
2012 Apr 1 17:44:14 w001 java: %WAAS-CMS-4-716059: ce(DataFeedPoll): Module thread failed
2012 Apr 1 17:44:14 w001 java: %WAAS-CMS-0-700002: ce(DataFeedPoll): java.lang.OutOfMemoryError: unable to create new native thread: java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method) at java.lang.Thread.start(Unknown Source) at java.lang.UNIXProcess$1.run(Unknown Source) at java.security.AccessController.doPrivileged(Native Method) at java.lang.UNIXProcess.<init>(Unknown Source) at java.lang.ProcessImpl.start(Unknown Source) at java.lang.ProcessBuilder.start(Unknown Source) at java.lang.Runtime.exec(Unknown Source) at java.lang.Runtime.exec(Unknown Source) at java.lang.Runtime.exec(Unknown Source) at com.cisco.unicorn.util.Utils.SystemExec(Utils.java:781) at com.cisco.unicorn.pcm.LicenseManagementHook.execCli(LicenseManagementHook.java:121) at com.cisco.unicorn.pcm.LicenseManagementHook.parseLicenseNames(LicenseManagementHook.java:265) at com.cisco.unicorn.pcm.LicenseManagementHook.getLicenseChanges(LicenseManagementHook.java:380) at com.cisco.un
2012 Apr 2 11:08:35 w001 login: %WAAS-SYSUTL-5-800003: admin login on 0 from 10.72.131.102
2012 Apr 2 11:32:46 w001 login: %WAAS-SYSUTL-5-800003: admin login on 0 from 10.72.10.140
2012 Apr 2 11:36:20 w001 Nodemgr: %WAAS-NODEMGR-5-330082: [pmgr] requests stop service cms_ce.
2012 Apr 2 11:36:20 w001 Nodemgr: %WAAS-NODEMGR-5-330032: Stopping service: 'cms_ce'.
2012 Apr 2 11:36:20 w001 Nodemgr: %WAAS-NODEMGR-5-330036: Stop service 'cms_ce' using: '/sw/merlot/bin/spen ctl --stop --name ce' with pid: 10011
2012 Apr 2 11:36:21 w001 perl: %WAAS-CMS-4-700001: Timed out while trying to stop cms service.
2012 Apr 2 11:36:23 w001 Nodemgr: %WAAS-NODEMGR-5-330027: Process with pid 6890 exits
2012 Apr 2 11:36:40 w001 Nodemgr: %WAAS-NODEMGR-5-330080: [pmgr] requests start service cms_ce
2012 Apr 2 11:36:40 w001 Nodemgr: %WAAS-NODEMGR-5-330040: Start service 'cms_ce' using: '/sw/merlot/bin/spen ctl --start --name ce' with pid: 10146
2012 Apr 2 11:36:47 w001 java: %WAAS-CMS-4-700001: Creating new entry in the tcp_values_map for application Other
2012 Apr 2 11:36:47 w001 java: %WAAS-CMS-4-700001: Creating new entry in the tcp_values_map for application SQL
2012 Apr 2 11:36:47 w001 java: %WAAS-CMS-4-700001: Creating new entry in the tcp_values_map for application File-System
2012 Apr 2 11:36:47 w001 java: %WAAS-CMS-4-700001: Creating new entry in the tcp_values_map for application Systems-Management
2012 Apr 2 11:36:47 w001 java: %WAAS-CMS-4-700001: Creating new entry in the tcp_values_map for application Console
2012 Apr 2 11:36:47 w001 java: %WAAS-CMS-4-700001: Creating new entry in the tcp_values_map for application Remote-Desktop
2012 Apr 2 11:36:47 w001 java: %WAAS-CMS-4-700001: Creating new entry in the tcp_values_map for application Directory-Services
2012 Apr 2 11:36:47 w001 java: %WAAS-CMS-4-700001: Creating new entry in the tcp_values_map for application SSL
2012 Apr 2 11:36:47 w001 java: %WAAS-CMS-4-700001: Creating new entry in the tcp_values_map for application Version-Management
2012 Apr 2 11:36:47 w001 java: %WAAS-CMS-4-700001: Creating new entry in the tcp_values_map for application Conferencing
2012 Apr 2 11:36:47 w001 java: %WAAS-CMS-4-700001: Creating new entry in the tcp_values_map for application P2P
2012 Apr 2 11:36:47 w001 java: %WAAS-CMS-4-700001: Creating new entry in the tcp_values_map for application WAFS
2012 Apr 2 11:36:47 w001 java: %WAAS-CMS-4-700001: Creating new entry in the tcp_values_map for application Streaming
2012 Apr 2 11:36:47 w001 java: %WAAS-CMS-4-700001: Creating new entry in the tcp_values_map for application Email-and-Messaging
2012 Apr 2 11:36:47 w001 java: %WAAS-CMS-4-700001: Creating new entry in the tcp_values_map for application Enterprise-Applications
2012 Apr 2 11:36:47 w001 java: %WAAS-CMS-4-700001: Creating new entry in the tcp_values_map for application Printing
2012 Apr 2 11:36:47 w001 java: %WAAS-CMS-4-700001: Creating new entry in the tcp_values_map for application CAD
2012 Apr 2 11:36:47 w001 java: %WAAS-CMS-4-700001: Creating new entry in the tcp_values_map for application Authentication
2012 Apr 2 11:36:47 w001 java: %WAAS-CMS-4-700001: Creating new entry in the tcp_values_map for application File-Transfer
2012 Apr 2 11:36:47 w001 java: %WAAS-CMS-4-700001: Creating new entry in the tcp_values_map for application Backup
2012 Apr 2 11:36:47 w001 java: %WAAS-CMS-4-700001: Creating new entry in the tcp_values_map for application Call-Management
2012 Apr 2 11:36:47 w001 java: %WAAS-CMS-4-700001: Creating new entry in the tcp_values_map for application Web
2012 Apr 2 11:36:47 w001 java: %WAAS-CMS-4-700001: Creating new entry in the tcp_values_map for application Replication
2012 Apr 2 11:36:47 w001 java: %WAAS-CMS-4-700001: Creating new entry in the tcp_values_map for application Content-Management
2012 Apr 2 11:36:47 w001 java: %WAAS-CMS-4-700001: Creating new entry in the tcp_values_map for application Instant-Messaging
2012 Apr 2 11:36:47 w001 java: %WAAS-CMS-4-700001: Creating new entry in the tcp_values_map for application VPN
2012 Apr 2 11:36:47 w001 java: %WAAS-CMS-4-700001: Creating new entry in the tcp_values_map for application SSH
2012 Apr 2 11:36:47 w001 java: %WAAS-CMS-4-700001: Creating new entry in the tcp_values_map for application Name-Services
2012 Apr 2 11:36:47 w001 java: %WAAS-CMS-4-700001: Creating new entry in the tcp_values_map for application Storage
Best regards,
Eiji Ide
