LMS 3.2 - RME System Inventory Polling takes very long

lo.mueller · ‎06-11-2010

Hi there,

Environment: Solaris 10 (approx. 1400 devices managed)

With scheduled RME System Inventory Polling Job at 01:00 am it ends 6,5 hours later.

Looking into the Job-Log after about 53 minutes following messages are shown until job finish

[ Fri Jun 11 01:53:08 CEST 2010 ],INFO ,[Thread-2],Updating result for the device: SW-5941
[ Fri Jun 11 06:38:10 CEST 2010 ],INFO ,[Timer-0], This timer task is not valid anymore, exiting.
[ Fri Jun 11 06:38:44 CEST 2010 ],INFO ,[Timer-0], This timer task is not valid anymore, exiting.
[ Fri Jun 11 06:38:44 CEST 2010 ],INFO ,[Timer-0], This timer task is not valid anymore, exiting.
[ Fri Jun 11 06:38:44 CEST 2010 ],INFO ,[Timer-0], This timer task is not valid anymore, exiting.
[ Fri Jun 11 06:38:45 CEST 2010 ],INFO ,[Timer-0], This timer task is not valid anymore, exiting.
[ Fri Jun 11 06:38:47 CEST 2010 ],INFO ,[Timer-0], This timer task is not valid anymore, exiting.
[ Fri Jun 11 06:38:51 CEST 2010 ],INFO ,[Timer-0], This timer task is not valid anymore, exiting.
[ Fri Jun 11 06:38:55 CEST 2010 ],INFO ,[Timer-0], This timer task is not valid anymore, exiting.

Last entries in IC_Server.log

[ Fri Jun 11 01:53:08 CEST 2010 ],INFO ,[Thread-19],com.cisco.nm.rmeng.inventory.ics.core.CollectionController,596,No Trap
[ Fri Jun 11 01:53:08 CEST 2010 ],INFO ,[Thread-19],com.cisco.nm.rmeng.inventory.ics.core.CollectionController,602,CALLING SELF TEST
[ Fri Jun 11 01:53:08 CEST 2010 ],INFO ,[Thread-19],com.cisco.nm.rmeng.inventory.ics.core.CollectionController,604,DONE WITH SELF TEST
[ Fri Jun 11 01:53:08 CEST 2010 ],INFO ,[Thread-19],com.cisco.nm.rmeng.inventory.ics.core.CollectionController,492, Finished processing device 2246 10.222.68.5
[ Fri Jun 11 01:53:08 CEST 2010 ],INFO ,[Thread-19],com.cisco.nm.rmeng.inventory.ics.core.CollectionController,498,Total time is 28 seconds for 10.222.68.5

Please find attached log files (excerpt).

The Job finishes with failure status and shows following results

Failed: 19

Not Attempted: 1

Successful With changes: 628

There seems to be a lack of over 700 devices with Inventory Polling.

Does anybody have an idea?

Thanks for you feedback

Lothar

Joe Clarke · ‎06-12-2010

With only excerpts of logs and no debugging, it's impossible to say what the problem is. I don't see anything here that shows a problem. Enable ICServer debugging under RME > Admin > System Preferences > Application Loglevel Settings. Then, if possible, reschedule the system polling job to run while you are there. It is important to be able to capture a thread dump of the job's JVM as well as ICServer's JVM when the output in the job log stops. To capture these dumps, send a SIGQUIT to the PIDs of ICServer and the job. You can get both PIDs from the output of the pdshow command. The job's process name will be the job ID number.