LMS 2.6: ICServer crash?

yjdabear · ‎08-23-2007

I was checking to see why the Inventory Collection "failed" after scanning only 48 devices, even though no specific device was shown as "failed".

Start Time: Aug 22 2007 22:33:05 EDT

End Time: Aug 23 2007 04:06:37 EDT

Duration: 5 hours, 33 minutes, 30.56 seconds

Average Scan Time: 6 minutes, 56.876 seconds/device

Job Result Description: Timed out. Make sure that ICServer is running.

pdshow output:

Process= ICServer

State = Administrator has shut down this server

Pid = 0

RC = 1

Signo = 0

Start = 08/20/07 01:14:39

Stop = 08/22/07 22:37:40

Core = Not applicable

Info = ICServer started.

tail IC_Server.log

[ Wed Aug 22 22:36:37 EDT 2007 ],INFO ,[Thread-20],com.cisco.nm.rmeng.inventory.ics.core.CollectionController,194,Started processing device Name: 172.18.23.12

[ Wed Aug 22 22:36:38 EDT 2007 ],WARN ,[Thread-14],com.cisco.nm.rmeng.util.logger.XDILogger,67,com.cisco.nm.xms.xdi.pkgs.SharedInventoryCatSwitch.ContainmentAGI_STACK_Mib,getModuleValues,1201,Illegal Access. NoSuchInstance being accessed as Integer.

[ Wed Aug 22 22:36:38 EDT 2007 ],INFO ,[Thread-14],com.cisco.nm.rmeng.util.logger.XDILogger,77,com.cisco.nm.xms.xdi.pkgs.SharedInventoryCatSwitch.ImageAGI_CISCO_IMAGE_Mib,g$eval,62,com.cisco.nm.xms.xdi.pkgs.SharedInventoryCatSwitch:ImageAGI_CISCO_IMAGE_Mib:g$eval:populating ImageAG attributes, begins...

[ Wed Aug 22 22:36:41 EDT 2007 ],INFO ,[Thread-14],com.cisco.nm.rmeng.util.logger.XDILogger,77,com.cisco.nm.xms.xdi.pkgs.SharedInventoryCatSwitch.ImageAGI_CISCO_IMAGE_Mib,g$eval,71,com.cisco.nm.xms.xdi.pkgs.SharedInventoryCatSwitch:ImageAGI_CISCO_IMAGE_Mib:g$eval:populatingImageAG attributes, completed

[ Wed Aug 22 22:36:44 EDT 2007 ],INFO ,[Thread-13],com.cisco.nm.rmeng.inventory.ics.core.CollectionController,239,DP time is 32 seconds for xx.xx.xx.xx

[ Wed Aug 22 22:36:58 EDT 2007 ],INFO ,[Thread-17],com.cisco.nm.rmeng.inventory.ics.core.CollectionController,239,DP time is 40 seconds for xx.xx.xx.xx

[ Wed Aug 22 22:37:28 EDT 2007 ],FATAL,[Thread-13],?,?,Fatal Error has Occured, exiting ICServer java.lang.OutOfMemoryError

[ Wed Aug 22 22:37:34 EDT 2007 ],INFO ,[Thread-13],com.cisco.nm.rmeng.inventory.ics.server.InvDataProcessor,482,Total Memory :132907008

[ Wed Aug 22 22:37:34 EDT 2007 ],INFO ,[Thread-13],com.cisco.nm.rmeng.inventory.ics.server.InvDataProcessor,483,Free Memory :564432

[ Wed Aug 22 22:37:34 EDT 2007 ],INFO ,[Thread-13],com.cisco.nm.rmeng.inventory.ics.server.InvDataProcessor,484,Max Memory :132907008

It looks like ICServer has crashed. Is this a softwarre of hardware issue or a one-time anormaly?

David Stanford · ‎08-23-2007

How much RAM in this server?

you might want to try increasing the heap size:

1. Please, stop Daemon Manager, issuing the following command:

# /etc/init.d/dmgtd stop

2. Please wait 10 minutes. After that, make sure that no LMS process is

running (you can check it by issuing a "ps -ef | grep CSCOpx" command).

3. Make a backup of the following file, creating a dmgtd.conf.bak file

in some other folder in your server:

/opt/CSCOpx/objects/dmgt/dmgtd.conf

4. Edit the /opt/CSCOpx/objects/dmgt/dmgtd.conf file and search for the

ICServer entry

To increase the heap size, please update

/opt/CSCOpx/objects/dmgt/dmgtd.conf. Inside this file search for

ICServer. It will be something like

/opt/CSCOpx/bin/cwjava ... -Xmx128m

...com.cisco.nm.rmeng.inventory.ics.server.ICServer

Change the -Xmx128m value to -Xmx256m to double it and save the file.

5. Start Damon Manager and wait for 10 minutes in order to let all the

processes to start properly.

yjdabear · ‎08-23-2007

There's 16GB of RAM. The server is shared with another app with an Oracle db backend.

I won't be able to try this right away, but looking at the section you specified in dmgtd.conf, there's also an "-Xms128m". Does that need to be changed as well? Are there any published guides on tuning such parameters in dmgtd.conf?

ICServer y n 0 EssMonitor,RMEDbMonitor,RMECSTMServer /opt/CSCOpx/bin/cwjava -cw

/opt/CSCOpx -cw:jre lib/jre -server -Xms128m -Xmx128m -cp:p MDC/tomcat/shared/li

David Stanford · ‎08-23-2007

More specific, here's what you'll want to change:

For Args variable change the value from:

-Xms128m -Xmx128m

To

-Xms256m -Xmx256m

change 128 to 256 in each instance