12-29-2009 07:47 AM
Background: LMS 3.1 ( CS 3.2,CM 5.1.3,RME 4.2,portal 1.1 )
OS: Windows 2003 SP2.
Had windows patches installed one week earlier.
Remoted (RDP) into server for daily checks and found CPU at 100%
This has happened before when our discovery took to long. Assuming the same issue, I restarted the services.
As they came up, they took several minutes longer than normal and seemed to be stalled when Tomcat started.
All eventually show as started except for apache.
I checked the event viewer for CASUSER which had successful audits.
My current sll dll's are dated 1/18/2008
I checked all the user rights according to a previous post.
I added CASUSER to the administrators group.
and finally thinking it may have bee a corrupted databse I did a restore and still have the same symptom.
Symptom: It looks like Tomcat isn't coming up all the way which never starts Tomcat monitor and therefore no apache...
I will appologize in advance for slow responses as I am dealing with a completely isolated network and may have to recreate
sections of output that you ask for. I attached pdreg-l appache and pdshow....
Request: Any suggestions are appreciated...... I also recently learned about the points system so I will try to vote on my previous threads.
12-29-2009 09:11 AM
You did not post your pdshow or pdreg output. You should also post screenshots of the Services control panel showing all of the CiscoWorks services.
12-29-2009 09:27 AM
12-29-2009 09:42 AM
Post the NMSROOT/MDC/tomcat/logs/stdout.log and stderr.log. What other software do you have installed on this server?
12-29-2009 10:12 AM
The files are attached...
stdout was empty 0k
Applications on the server beside CiscoWorks:
BMC patrol (AGENT)
Veritas Netbackup
Symantec Client Security (Antivirus)
Arcsight smart agent
When first troubleshooting, I did notice that the antivirus was scanning the CSCOpx directories. This had previously been
excluded and is again but I don't know if it could have done any damage.
12-29-2009 11:07 AM
Do you see any hs_err* files on the file system? If so, post them.
12-29-2009 11:37 AM
12-29-2009 11:49 AM
Post your NMSROOT/MDC/etc/regdaemon.xml file.
12-29-2009 12:09 PM
I appologize for not being able to provide the file you need.
I am not able to move files off the system. Would it benefit me to compare it to the same file on an existing working server?
or provide a particular section of the file?
I understand if you are unable to go any further without this data. Given that TAC will probably want the same files,
do you think I am better off uninstalling LMS, reinstalling and trying to restore a backup?
Thanks for all the help!
12-29-2009 12:25 PM
It is important to see the entire regdaemon.xml file as it exists on the server. If that file is corrupt, then restoring a backup which contains it would simply reintroduce the same problem.
Given what I have seen thus far, I cannot say what is causing your problem. A reinstall may or may not fix it. It could be due to a corrupt regdaemon.xml, or it might be some problem with anti-virus interaction. There could also be some corruption of the Tomcat config files. It would be easy to verify the former and the latter if you could provide the files. The issue with AV could be corrected by a reininstall provided AV was disabled, then on-access scanning remained disabled for NMSROOT.
Of course, this may be something else entirely. The only thing that's clear is that Tomcat is not starting properly, and appears to be hanging (or maybe looping) trying to load the Cisco components of LMS. It may boil down to requiring remote access to look at the problem in real time.
12-29-2009 12:41 PM
I sooo wish I could allow remote access....
I will see what options I have for providing the files... I can tell you that the regdeamon file appears different from
a June regdaemon.xml.bak in the same folder as the order of the
Would it hurt to copy the older regdaemon.xml.bak of the existing one ?
As far as the Anti Virus, are you saying I would need to reinstall CiscoWorks?
12-29-2009 01:11 PM
Without validating the current regdaemon.xml, I cannot recommend doing anything. The current file may be fine, in which case restoring the old file could break something else. If the problem is that the AV software has somehow broken the LMS jar files, or messed with the Tomcat config, then a reinstall would correct that.
12-30-2009 04:51 AM
It sounds like circumstances are going to force me to reinstall. I guess at that point I could try the old regdaemon.xml file first.
Will I have to do a complete uninstall / install ? And if so will I be able to restore a backup after that?
12-30-2009 09:47 AM
I cannot answer these questions with certainty. If you're going to reinstall, you should reinstall from scratch. To be 100% safe, you should obtain the clean_system.exe tool from TAC which will make sure the system is pristine before doing the reinstall.
The backup MAY work provided there is no corruption in the LMS data (and I cannot say for certain if corruption exists now). My advice is to do the clean reinstall. When you have it installed, test it before doing any restore. Make sure you can login. Try adding one device, and make sure things work. Then, take a backup of LMS at that state. Then try the restore. If things are still working, go forth in peace. If not, you should be able to restore the pristine backup, and start over (without doing another reinstall).
01-03-2010 07:29 AM
JClarke,
I did try taking a regdaemon.xml file form a similarly configured box (paths/os..etc), updated the unique fields, (server name, ACS key..etc)
and starting the services with the same result. I will take one last shot at getting the regdaemon.xml file to you but will also start planning
for a reinstall. again appreciate all the help....I'll keep you posted.
Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: