CER 8.6.1 - Host Resources Agent - Stopped / Won't Start

[Kirill Klimakhin]

The Host Resources Agent service on our 8.6.1 CER server has stopped, and thus is no longer SNMP pollable.

Whenever I try to start the service, it says starting, never starts and stays stopped.

I rebooted the server and the service is still down.

Any ideas?

[yahsiel2004]

Have you checked the event viewer logs under Serviceability?

HTH

Regards,

Yosh

[Kirill Klimakhin]

Here is what I am seeing in there:

database

2014-02-06 02:21:44,734 | INFO  | CER_DATABASE | Failed to get the fully qualified host name. DNS might not be enabled. Putting Ipaddress in Cluster database

2014-02-06 04:21:59,100 | INFO  | CER_DATABASE | Failed to get the fully qualified host name. DNS might not be enabled. Putting Ipaddress in Cluster database

2014-02-06 06:22:18,522 | INFO  | CER_DATABASE | Failed to get the fully qualified host name. DNS might not be enabled. Putting Ipaddress in Cluster database

2014-02-06 08:22:30,544 | INFO  | CER_DATABASE | Failed to get the fully qualified host name. DNS might not be enabled. Putting Ipaddress in Cluster database

2014-02-06 09:52:40,244 | INFO  | CER_DATABASE | Failed to get the fully qualified host name. DNS might not be enabled. Putting Ipaddress in Cluster database

2014-02-06 11:52:53,872 | INFO  | CER_DATABASE | Failed to get the fully qualified host name. DNS might not be enabled. Putting Ipaddress in Cluster database

2014-02-06 13:53:08,475 | INFO  | CER_DATABASE | Failed to get the fully qualified host name. DNS might not be enabled. Putting Ipaddress in Cluster database

2014-02-06 14:23:12,969 | INFO  | CER_DATABASE | Failed to get the fully qualified host name. DNS might not be enabled. Putting Ipaddress in Cluster database

2014-02-06 15:53:23,381 | INFO  | CER_DATABASE | Failed to get the fully qualified host name. DNS might not be enabled. Putting Ipaddress in Cluster database

2014-02-06 16:47:56,014 | WARN  | CER_GROUP | Disconnected from Cisco Emergency Responder : Default_DE / /10.200.0.13

2014-02-06 16:47:56,015 | INFO  | CER_CALLENGINE | Cisco ER Exiting : Graceful shutdown

2014-02-06 16:56:49,317 | INFO  | CER_GROUP | Active Cisco Emergency Responder set to : Publisher / CER/10.200.0.13

2014-02-06 16:57:22,836 | INFO  | CER_GROUP | Connection established with Cisco Emergency Responder : Default_DE / /10.200.0.13

2014-02-06 16:57:23,001 | INFO  | CER_GROUP | Connection established with Cisco Emergency Responder : Aggregator / /10.200.0.13

2014-02-06 16:57:24,094 | WARN  | CER_TELEPHONY | 911Route Point received OUT_SERVICE event from Provider 10.200.0.10

2014-02-06 16:58:30,259 | INFO  | CER_DATABASE | Failed to get the fully qualified host name. DNS might not be enabled. Putting Ipaddress in Cluster database

2014-02-06 16:59:39,265 | INFO  | CER_DATABASE | Failed to get the fully qualified host name. DNS might not be enabled. Putting Ipaddress in Cluster database

2014-02-06 17:00:33,156 | WARN  | CER_TELEPHONY | 911Route Point received OUT_SERVICE event from Provider 10.200.0.10

[yahsiel2004]

What about under System Logs>Platform Logs>Syslog>CiscoSyslog?

HTH

Regards,

Yosh

[Kirill Klimakhin]

Here is what I am seeing:

Feb  6 16:34:24 CER local7 2 : 18: CER: Feb 06 2014 21:34:24.937 UTC :  %UC_SERVICEMANAGER-2-ServiceStartFailed: %[ServiceName=Host Resources Agent][Reason=Unsuccessful completion][AppID=Cisco Service Manager][ClusterID=][NodeID=CER]: Failed to start service.

Feb  6 16:34:24 CER local7 2 : 19: CER: Feb 06 2014 21:34:24.937 UTC :  %UC_SERVICEMANAGER-2-ServiceFailed: %[ServiceName=Host Resources Agent][Reason=Service stopped abruptly][AppID=Cisco Service Manager][ClusterID=][NodeID=CER]: Service terminated.

Feb  6 16:35:39 CER local7 2 : 20: CER: Feb 06 2014 21:35:39.93 UTC :  %UC_SERVICEMANAGER-2-ServiceStartFailed: %[ServiceName=Host Resources Agent][Reason=Unsuccessful completion][AppID=Cisco Service Manager][ClusterID=][NodeID=CER]: Failed to start service.

Feb  6 16:35:39 CER local7 2 : 21: CER: Feb 06 2014 21:35:39.93 UTC :  %UC_SERVICEMANAGER-2-ServiceFailed: %[ServiceName=Host Resources Agent][Reason=Service stopped abruptly][AppID=Cisco Service Manager][ClusterID=][NodeID=CER]: Service terminated.

Feb  6 16:36:55 CER local7 2 : 22: CER: Feb 06 2014 21:36:55.229 UTC :  %UC_SERVICEMANAGER-2-ServiceStartFailed: %[ServiceName=Host Resources Agent][Reason=Unsuccessful completion][AppID=Cisco Service Manager][ClusterID=][NodeID=CER]: Failed to start service.

Feb  6 16:36:55 CER local7 2 : 23: CER: Feb 06 2014 21:36:55.229 UTC :  %UC_SERVICEMANAGER-2-ServiceFailed: %[ServiceName=Host Resources Agent][Reason=Service stopped abruptly][AppID=Cisco Service Manager][ClusterID=][NodeID=CER]: Service terminated.

Feb  6 16:37:14 CER local7 2 : 24: CER: Feb 06 2014 21:37:14.322 UTC :  %UC_SERVICEMANAGER-2-ServiceFailed: %[ServiceName=Host Resources Agent][Reason=Service stopped abruptly][AppID=Cisco Service Manager][ClusterID=][NodeID=CER]: Service terminated.

Feb  6 16:37:14 CER local7 1 : 25: CER: Feb 06 2014 21:37:14.323 UTC :  %UC_SERVICEMANAGER-1-ServiceExceededMaxRestarts: %[ServiceName=Host Resources Agent][RestartCount=3][AppID=Cisco Service Manager][ClusterID=][NodeID=CER]: Service failed to start and exceeded maximum allowed restarts.

Feb  6 16:54:56 CER local7 3 : Alarm Interface bind failed for catalog: 6 GenericAlarmCatalog

Feb  6 16:54:56 CER local7 3 : Unknown Alarm Catalog 32 GenericAlarmCatalog

Feb  6 16:54:59 CER local7 3 : Alarm Interface bind failed for catalog: 6 GenericAlarmCatalog

Feb  6 16:54:59 CER local7 3 : Unknown Alarm Catalog 32 GenericAlarmCatalog

Feb  6 16:56:10 CER local7 3 : Alarm Interface bind failed for catalog: 6 CDPAlarmCatalog

Feb  6 16:56:10 CER local7 3 : Unknown Alarm Catalog 32 CDPAlarmCatalog

Feb  6 16:56:27 CER local7 3 : Alarm Interface bind failed for catalog: 6 JavaApplications

Feb  6 16:56:31 CER local7 3 : Unknown Alarm Catalog 32 JavaApplications

Feb  6 16:57:23 CER local7 2 : 0: CER: Feb 06 2014 21:57:23.590 UTC :  %UC_SERVICEMANAGER-2-ServiceStartFailed: %[ServiceName=Host Resources Agent][Reason=Unsuccessful completion][AppID=Cisco Service Manager][ClusterID=][NodeID=CER]: Failed to start service.

Feb  6 16:57:23 CER local7 2 : 1: CER: Feb 06 2014 21:57:23.590 UTC :  %UC_SERVICEMANAGER-2-ServiceFailed: %[ServiceName=Host Resources Agent][Reason=Service stopped abruptly][AppID=Cisco Service Manager][ClusterID=][NodeID=CER]: Service terminated.

Feb  6 16:58:44 CER local7 2 : 2: CER: Feb 06 2014 21:58:44.809 UTC :  %UC_SERVICEMANAGER-2-ServiceStartFailed: %[ServiceName=Host Resources Agent][Reason=Unsuccessful completion][AppID=Cisco Service Manager][ClusterID=][NodeID=CER]: Failed to start service.

Feb  6 16:58:44 CER local7 2 : 3: CER: Feb 06 2014 21:58:44.809 UTC :  %UC_SERVICEMANAGER-2-ServiceFailed: %[ServiceName=Host Resources Agent][Reason=Service stopped abruptly][AppID=Cisco Service Manager][ClusterID=][NodeID=CER]: Service terminated.

Feb  6 17:00:04 CER local7 2 : 4: CER: Feb 06 2014 22:00:04.918 UTC :  %UC_SERVICEMANAGER-2-ServiceStartFailed: %[ServiceName=Host Resources Agent][Reason=Unsuccessful completion][AppID=Cisco Service Manager][ClusterID=][NodeID=CER]: Failed to start service.

Feb  6 17:00:04 CER local7 2 : 5: CER: Feb 06 2014 22:00:04.919 UTC :  %UC_SERVICEMANAGER-2-ServiceFailed: %[ServiceName=Host Resources Agent][Reason=Service stopped abruptly][AppID=Cisco Service Manager][ClusterID=][NodeID=CER]: Service terminated.

Feb  6 17:01:25 CER local7 2 : 6: CER: Feb 06 2014 22:01:25.702 UTC :  %UC_SERVICEMANAGER-2-ServiceStartFailed: %[ServiceName=Host Resources Agent][Reason=Unsuccessful completion][AppID=Cisco Service Manager][ClusterID=][NodeID=CER]: Failed to start service.

Feb  6 17:01:25 CER local7 2 : 7: CER: Feb 06 2014 22:01:25.703 UTC :  %UC_SERVICEMANAGER-2-ServiceFailed: %[ServiceName=Host Resources Agent][Reason=Service stopped abruptly][AppID=Cisco Service Manager][ClusterID=][NodeID=CER]: Service terminated.

Feb  6 17:01:25 CER local7 1 : 8: CER: Feb 06 2014 22:01:25.704 UTC :  %UC_SERVICEMANAGER-1-ServiceExceededMaxRestarts: %[ServiceName=Host Resources Agent][RestartCount=3][AppID=Cisco Service Manager][ClusterID=][NodeID=CER]: Service failed to start and exceeded maximum allowed restarts.

[yahsiel2004]

Have you checked your current "Processes" to make sure that you don't have a service that is pegging your CPU. I would verify the "Processes" first and then if nothing seems excessive I would stop all the services and try to just start the "Host Resources Agent" service.

You could also check the "boot log" it might give a pointer in the right direction.

HTH

Regards,

Yosh

[Kirill Klimakhin]

Tomcat is the highest consumer, at 1.1%

I am going to try bringing down the services now. Thank you for all your suggestions Yosh.

[yahsiel2004]

You're welcome, I know CER is a tricky one to troubleshoot, since its so different from all the rest.

Regards,

Yosh

[Kirill Klimakhin]

Yosh, shut down all the services (all but some that wouldn't not allow me to bring them down for obvious reasons) and tried to fire up the host resource server but it just looped like it always does and then fails to start after maximum tries.

Guess it's time for a TAC case?

[yahsiel2004]

Sounds like it. I'm curious to know what the issue is, keep me posted on solution.

Regards,

Yosh

[Kirill Klimakhin]

Yosh - here is the problem:

https://tools.cisco.com/bugsearch/bug/CSCtw73440

[yahsiel2004]

I appreciate the follow up. So this has solved your issue?

Regards,

Yosh

[Kirill Klimakhin]

Sorry just saw this - yes the COP file provided by TAC solved the issue.

[yahsiel2004]

Good to know.

Thanks,

Yosh