LMS 2.6: ICServer crash?

Unanswered Question
Aug 23rd, 2007
User Badges:
  • Gold, 750 points or more

I was checking to see why the Inventory Collection "failed" after scanning only 48 devices, even though no specific device was shown as "failed".


Start Time: Aug 22 2007 22:33:05 EDT

End Time: Aug 23 2007 04:06:37 EDT


Duration: 5 hours, 33 minutes, 30.56 seconds

Average Scan Time: 6 minutes, 56.876 seconds/device

Job Result Description: Timed out. Make sure that ICServer is running.


pdshow output:


Process= ICServer

State = Administrator has shut down this server

Pid = 0

RC = 1

Signo = 0

Start = 08/20/07 01:14:39

Stop = 08/22/07 22:37:40

Core = Not applicable

Info = ICServer started.


tail IC_Server.log

[ Wed Aug 22 22:36:37 EDT 2007 ],INFO ,[Thread-20],com.cisco.nm.rmeng.inventory.ics.core.CollectionController,194,Started processing device Name: 172.18.23.12

[ Wed Aug 22 22:36:38 EDT 2007 ],WARN ,[Thread-14],com.cisco.nm.rmeng.util.logger.XDILogger,67,com.cisco.nm.xms.xdi.pkgs.SharedInventoryCatSwitch.ContainmentAGI_STACK_Mib,getModuleValues,1201,Illegal Access. NoSuchInstance being accessed as Integer.

[ Wed Aug 22 22:36:38 EDT 2007 ],INFO ,[Thread-14],com.cisco.nm.rmeng.util.logger.XDILogger,77,com.cisco.nm.xms.xdi.pkgs.SharedInventoryCatSwitch.ImageAGI_CISCO_IMAGE_Mib,g$eval,62,com.cisco.nm.xms.xdi.pkgs.SharedInventoryCatSwitch:ImageAGI_CISCO_IMAGE_Mib:g$eval:populating ImageAG attributes, begins...

[ Wed Aug 22 22:36:41 EDT 2007 ],INFO ,[Thread-14],com.cisco.nm.rmeng.util.logger.XDILogger,77,com.cisco.nm.xms.xdi.pkgs.SharedInventoryCatSwitch.ImageAGI_CISCO_IMAGE_Mib,g$eval,71,com.cisco.nm.xms.xdi.pkgs.SharedInventoryCatSwitch:ImageAGI_CISCO_IMAGE_Mib:g$eval:populatingImageAG attributes, completed

[ Wed Aug 22 22:36:44 EDT 2007 ],INFO ,[Thread-13],com.cisco.nm.rmeng.inventory.ics.core.CollectionController,239,DP time is 32 seconds for xx.xx.xx.xx

[ Wed Aug 22 22:36:58 EDT 2007 ],INFO ,[Thread-17],com.cisco.nm.rmeng.inventory.ics.core.CollectionController,239,DP time is 40 seconds for xx.xx.xx.xx

[ Wed Aug 22 22:37:28 EDT 2007 ],FATAL,[Thread-13],?,?,Fatal Error has Occured, exiting ICServer java.lang.OutOfMemoryError

[ Wed Aug 22 22:37:34 EDT 2007 ],INFO ,[Thread-13],com.cisco.nm.rmeng.inventory.ics.server.InvDataProcessor,482,Total Memory :132907008

[ Wed Aug 22 22:37:34 EDT 2007 ],INFO ,[Thread-13],com.cisco.nm.rmeng.inventory.ics.server.InvDataProcessor,483,Free Memory :564432

[ Wed Aug 22 22:37:34 EDT 2007 ],INFO ,[Thread-13],com.cisco.nm.rmeng.inventory.ics.server.InvDataProcessor,484,Max Memory :132907008


It looks like ICServer has crashed. Is this a softwarre of hardware issue or a one-time anormaly?

  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 5 (1 ratings)
Loading.
David Stanford Thu, 08/23/2007 - 06:03
User Badges:
  • Cisco Employee,

How much RAM in this server?


you might want to try increasing the heap size:


1. Please, stop Daemon Manager, issuing the following command:

# /etc/init.d/dmgtd stop


2. Please wait 10 minutes. After that, make sure that no LMS process is

running (you can check it by issuing a "ps -ef | grep CSCOpx" command).


3. Make a backup of the following file, creating a dmgtd.conf.bak file

in some other folder in your server:

/opt/CSCOpx/objects/dmgt/dmgtd.conf


4. Edit the /opt/CSCOpx/objects/dmgt/dmgtd.conf file and search for the

ICServer entry


To increase the heap size, please update

/opt/CSCOpx/objects/dmgt/dmgtd.conf. Inside this file search for

ICServer. It will be something like


/opt/CSCOpx/bin/cwjava ... -Xmx128m

...com.cisco.nm.rmeng.inventory.ics.server.ICServer


Change the -Xmx128m value to -Xmx256m to double it and save the file.


5. Start Damon Manager and wait for 10 minutes in order to let all the

processes to start properly.


yjdabear Thu, 08/23/2007 - 06:15
User Badges:
  • Gold, 750 points or more

There's 16GB of RAM. The server is shared with another app with an Oracle db backend.


I won't be able to try this right away, but looking at the section you specified in dmgtd.conf, there's also an "-Xms128m". Does that need to be changed as well? Are there any published guides on tuning such parameters in dmgtd.conf?


ICServer y n 0 EssMonitor,RMEDbMonitor,RMECSTMServer /opt/CSCOpx/bin/cwjava -cw

/opt/CSCOpx -cw:jre lib/jre -server -Xms128m -Xmx128m -cp:p MDC/tomcat/shared/li

David Stanford Thu, 08/23/2007 - 06:26
User Badges:
  • Cisco Employee,

More specific, here's what you'll want to change:


For Args variable change the value from:


-Xms128m -Xmx128m


To


-Xms256m -Xmx256m


change 128 to 256 in each instance

Actions

This Discussion