JRM server is down or inaccessible for this user

Answered Question
Dec 15th, 2009

Hi,

We have recently starting getting this error message appearing on one of our Windows CiscoWorks servers. This server runs:

  • CS 3.0.6
  • RME 4.0.6
  • CiscoView 6.1.5

It is also the Master server in our environment.

The message started appearing after our OS Support team installed a round of MS Patches. I am wondering if this has started because they did not shutdown the CiscoWorks services properly first, using the net stop crmdmgtd command.

They have since removed the patches but the problem still exists.

Looking through the forums, I have noticed that other people have had similar problems in the past, though with slightly different scenario. I think I have checked all the possible solutions given, but still no joy.

The hostname has been changed since installation, but that was a couple of years ago - not recently. I have rechecked all the files that were involved in the rename and they still have the new name in them.

D:\CSCOpx\bin>pdshow CTMJrmServer
        Process= CTMJrmServer
        State  = Administrator has shut down this server
        Pid    = 0
        RC     = 0
        Signo  = 0
        Start  = 16/12/2009 12:29:01 p.m.
        Stop   = 16/12/2009 12:31:01 p.m.
        Core   = Not applicable
        Info   = Server started by admin request

The CTMJrmServer.lof file contains:

CTMJrmServer: initializing daemon manager
CTMJrmServer: initialized daemon manager
[ Wed Dec 16  12:24:51 NZDT 2009 ],INFO ,[main],com.cisco.nm.rmeng.util.logger.ServiceLogLevelChanger,notifyLevelChange,35,publishing urn for jrmw-log
[ Wed Dec 16  12:24:52 NZDT 2009 ],INFO ,[main],com.cisco.nm.rmeng.util.logger.ServiceLogLevelChanger,notifyLevelChange,42,published urn as jrmw-log-RMELogLevelChange
[ Wed Dec 16  12:25:30 NZDT 2009 ],FATAL,[main],com.cisco.nm.rmeng.jrmwrapper.server.JobManagerExt,getJRM,139,getJRM:Can't bind to JrmServiceManager. giving up and exiting.
[ Wed Dec 16  12:25:30 NZDT 2009 ],ERROR,[main],com.cisco.nm.rmeng.jrmwrapper.server.CTMJobManagerServer,<init>,66,Null JRM. Exiting
CTMJrmServer: initializing daemon manager
CTMJrmServer: initialized daemon manager
[ Wed Dec 16  12:27:00 NZDT 2009 ],INFO ,[main],com.cisco.nm.rmeng.util.logger.ServiceLogLevelChanger,notifyLevelChange,35,publishing urn for jrmw-log
[ Wed Dec 16  12:27:01 NZDT 2009 ],INFO ,[main],com.cisco.nm.rmeng.util.logger.ServiceLogLevelChanger,notifyLevelChange,42,published urn as jrmw-log-RMELogLevelChange
[ Wed Dec 16  12:27:32 NZDT 2009 ],FATAL,[main],com.cisco.nm.rmeng.jrmwrapper.server.JobManagerExt,getJRM,139,getJRM:Can't bind to JrmServiceManager. giving up and exiting.
[ Wed Dec 16  12:27:32 NZDT 2009 ],ERROR,[main],com.cisco.nm.rmeng.jrmwrapper.server.CTMJobManagerServer,<init>,66,Null JRM. Exiting
CTMJrmServer: initializing daemon manager
CTMJrmServer: initialized daemon manager
[ Wed Dec 16  12:29:01 NZDT 2009 ],INFO ,[main],com.cisco.nm.rmeng.util.logger.ServiceLogLevelChanger,notifyLevelChange,35,publishing urn for jrmw-log
[ Wed Dec 16  12:29:03 NZDT 2009 ],INFO ,[main],com.cisco.nm.rmeng.util.logger.ServiceLogLevelChanger,notifyLevelChange,42,published urn as jrmw-log-RMELogLevelChange
[ Wed Dec 16  12:29:33 NZDT 2009 ],FATAL,[main],com.cisco.nm.rmeng.jrmwrapper.server.JobManagerExt,getJRM,139,getJRM:Can't bind to JrmServiceManager. giving up and exiting.
[ Wed Dec 16  12:29:33 NZDT 2009 ],ERROR,[main],com.cisco.nm.rmeng.jrmwrapper.server.CTMJobManagerServer,<init>,66,Null JRM. Exiting

Needless to say that restarting the services, and the server have not had any success.

Anything else I can try?

Regards

Jeff

I have this problem too.
0 votes
Correct Answer by Joe Clarke about 6 years 10 months ago

Yes, do the master first, then update the slaves with the master's new hostname.  Don't forget to reaccept the SSL certificates on the slaves.

  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 5 (1 ratings)
Loading.
Joe Clarke Tue, 12/15/2009 - 17:40

Post the jrm.log, the output of the hostname command from the DOS prompt, and the output of "pdreg -l CTMJrmServer".

Jeff Law Tue, 12/15/2009 - 19:26

Thanks for your response.

The output of the pdreg -l CTMJrmServer is:

D:\CSCOpx\bin>pdreg -l CTMJrmServer
        Process      = CTMJrmServer
        Path         = D:\CSCOpx\bin\cwjava.exe
        Flags        = -cw D:\CSCOpx -cw:jre lib/jre -cp:p MDC/tomcat/webapps/rme/WEB-INF/classes,MDC/tomcat/webapps/rme/WEB-INF/lib/ctm.jar,MDC/tomcat/webapps/rme/WEB-INF/lib/log4j.jar com.cisco.nm.rmeng.jrmwrapper.server.CTMJobManagerServer geb
        Startup      = Started automatically at boot.
        Dependencies = RMEDbMonitor jrm TomcatMonitor RMECSTMServer

The output of the hostname command is:

D:\CSCOpx\bin>hostname
hp1813

The Jrm.log file is attached.

Attachment: 
Joe Clarke Tue, 12/15/2009 - 20:39

CTMJrmServer still thinks that the server's hostname is geb.  Was this the hostname that was changed a while back, or was the server's name changed again?

Jeff Law Tue, 12/15/2009 - 22:36

The server was originally built as HP1813 by the OS Team.

For various reasons, our team didnt want to call the device that, they wanted to call it GEB. This has caused more problems than it has solved as you can imagine.

So it was renamed after installation to GEB. Now by renaming, I probably mean aliasing. The computer name is still at HP1813 - so that the OS Team can manage and maintain it. We have DNS entries that refer to GEB and point to HP1813. The SSL Certificate is in the name of GEB.

There have been no changes regarding the naming of this server since installation a couple of years ago (that I know of ).

Should I go through the process that I followed for the renaming back then, again?

Regards

Jeff

Joe Clarke Tue, 12/15/2009 - 23:58

LMS is very particular about the hostname.  If you have changed the LMS files to point to geb, then I highly recommend you change the server's actual hostname to geb.  After you do this, reboot, then see if CTMJrmServer comes up.

Jeff Law Wed, 12/16/2009 - 13:05

I will see if we can get this organised - if only for a test, and not permanently.

Thanks

Jeff

Jeff Law Thu, 12/17/2009 - 13:44

We have renamed the server, and it appears to be working ok.

So, given that we have to rename the server back to its other name, we will have to work on changing the name ion CiscoWorks from GEB back to its other name.

I presume that I can follow the process I used to rename CiscoWorks originally, but in reverse. I will also have to generate some more SSL certificates in the new name.

Are there any issues that you can think of in doing this?

Regards

Jeff

Joe Clarke Thu, 12/17/2009 - 18:02

You will need to manually update the registry for CTMJrmServer to change geb to the actual hostname.  This is not properly documented in the hostname change guide.  But other than that, you shouldn't run into any issues.

Jeff Law Tue, 01/19/2010 - 16:49

Thanks for your reply. The holiday break is over, and it is back to work!

I will start the rename process off next week. Just two further questions....

Since we have 3 servers, with GEB the master one, should I shut down the others while doing the rename on GEB first, or will it not really matter?

I presume it would be better to do GEB before doing the other two?

Regards

Jeff

Correct Answer
Joe Clarke Sat, 01/23/2010 - 09:30

Yes, do the master first, then update the slaves with the master's new hostname.  Don't forget to reaccept the SSL certificates on the slaves.

Jeff Law Tue, 01/26/2010 - 17:38

I have organised for the server to be renamed back to its original name.

I have gone through the steps to remove any references to GEB in the CiscoWorks application on the master.

It took me a couple of goes to get the new certificate to be recognised, but managed that. Also had to disable, then reenable the SSO master role to get that bit working.

Unfortunately, I am still getting the orginal error again. The error I get now for the CTMJrmServer is :

H:\>pdshow CTMJrmServer
        Process= CTMJrmServer
        State  = Administrator has shut down this server
        Pid    = 0
        RC     = 0
        Signo  = 0
        Start  = 27/01/2010 2:28:42 p.m.
        Stop   = 27/01/2010 2:30:43 p.m.
        Core   = Not applicable
        Info   = Server started by admin request


H:\>

Below are the last entries in the CRMJrmServer.log file:

org.omg.CORBA.BAD_PARAM: Invalid GIOP Proxy ior: org.omg.CORBA.INV_OBJREF: Invalid object ref returned by URL resolver: exception com.inprise.vbroker.URLNaming.ReqFailure {
java.lang.String reason="java.io.FileNotFoundException: D:\CSCOpx\www\classpath\gatekeeper.ior (The system cannot find the file specified)"
}  minor code: 0  completed: No  minor code: 0  completed: No
at com.inprise.vbroker.firewall.FirewallLifeCycleInterceptor.getMechanismValue(FirewallLifeCycleInterceptor.java:139)
at com.inprise.vbroker.firewall.FirewallLifeCycleInterceptor.getMechanismValues(FirewallLifeCycleInterceptor.java:106)
at com.inprise.vbroker.firewall.FirewallLifeCycleInterceptor.create(FirewallLifeCycleInterceptor.java:59)
at com.inprise.vbroker.interceptor.ChainPOALifeCycleInterceptor.create(ChainPOALifeCycleInterceptor.java:22)
at com.inprise.vbroker.poa.POAImpl.(POAImpl.java:340)
at com.inprise.vbroker.poa.POAImpl.create_POA(POAImpl.java:727)
at com.cisco.nm.util.OrbUtils.createChildPOA(OrbUtils.java:486)
at com.cisco.nm.util.OrbUtils.createPersistentPOA(OrbUtils.java:477)
at com.cisco.nm.util.OrbUtils.createDefaultPOAs(OrbUtils.java:401)
at com.cisco.nm.util.OrbUtils.initMyORB(OrbUtils.java:381)
at com.cisco.nm.util.OrbUtils.initORB(OrbUtils.java:354)
at com.cisco.nm.rmeng.jrmwrapper.server.JobManagerExt.getJRM(JobManagerExt.java:95)
at com.cisco.nm.rmeng.jrmwrapper.server.JobManagerExt.(JobManagerExt.java:38)
at com.cisco.nm.rmeng.jrmwrapper.server.CTMJobManagerServer.(CTMJobManagerServer.java:60)
at com.cisco.nm.rmeng.jrmwrapper.server.CTMJobManagerServer.main(CTMJobManagerServer.java:34)
java.lang.Exception: com.cisco.nm.util.OrbUtils::Unable to initialize default POAs...
at com.cisco.nm.util.OrbUtils.initMyORB(OrbUtils.java:385)
at com.cisco.nm.util.OrbUtils.initORB(OrbUtils.java:354)
at com.cisco.nm.rmeng.jrmwrapper.server.JobManagerExt.getJRM(JobManagerExt.java:95)
at com.cisco.nm.rmeng.jrmwrapper.server.JobManagerExt.(JobManagerExt.java:38)
at com.cisco.nm.rmeng.jrmwrapper.server.CTMJobManagerServer.(CTMJobManagerServer.java:60)
at com.cisco.nm.rmeng.jrmwrapper.server.CTMJobManagerServer.main(CTMJobManagerServer.java:34)
[ Wed Jan 27  13:05:46 NZDT 2010 ],ERROR,[main],com.cisco.nm.rmeng.jrmwrapper.server.CTMJobManagerServer,,66,Null JRM. Exiting
CTMJrmServer: initializing daemon manager
CTMJrmServer: initialized daemon manager
[ Wed Jan 27  14:28:43 NZDT 2010 ],INFO ,[main],com.cisco.nm.rmeng.util.logger.ServiceLogLevelChanger,notifyLevelChange,35,publishing urn for jrmw-log
[ Wed Jan 27  14:28:43 NZDT 2010 ],INFO ,[main],com.cisco.nm.rmeng.util.logger.ServiceLogLevelChanger,notifyLevelChange,42,published urn as jrmw-log-RMELogLevelChange
[ Wed Jan 27  14:30:09 NZDT 2010 ],FATAL,[main],com.cisco.nm.rmeng.jrmwrapper.server.JobManagerExt,getJRM,139,getJRM:Can't bind to JrmServiceManager. giving up and exiting.
[ Wed Jan 27  14:30:09 NZDT 2010 ],ERROR,[main],com.cisco.nm.rmeng.jrmwrapper.server.CTMJobManagerServer,,66,Null JRM. Exiting

I dont understand the missing gatekeeper.ior file, as it does exist. Although as part of the steps I was following removed this file, it might have been created after this attempt to run.

The second entry in the log was when I manually tried to start the CRMJrmServer process.

Must be something else to change, but not sure what. Any further ideas welcome.

Also noticed that some of the other processes have not started.... Ive attached the output of pdshow.

Regards

Jeff

Jeff Law Tue, 01/26/2010 - 22:51

Well cant say I know exactly what I did, but the master server started respoding and behaving properly!!

I did notice that the jrm server was initialising for a while, which prevented other services from running, but that eventually worked. I did another restart and everything started working. Hopefully this will continue beyond the next restart.

I did notice while reading through the forums about jrm initializing, that this could be caused by anti virus products being able to scan the CSCOpx directory.

In our case this does happen, so I have requested that this directory, and all subdirectories, be excluded from any scanning. Hopefully that will help with the performance.

Once the master server was done, the others where pretty easy. I will check everything is still behaving itself tomorrow morning.

Thanks for all your help Joe. Much appreciated.

Regards

Jeff

Actions

This Discussion

Related Content