Solved: CiscoWorks Woes.

att0055rj · ‎03-21-2007

Hello,

I?ve posted on here a couple times about my CiscoWorks LMS 2.5 problems. The gist of what I was hearing was that people thought I should upgrade to LMS 2.6 and that would hopefully solve the issues

Previous post: http://forums.cisco.com/eforum/servlet/NetProf?page=netprof&CommCmd=MB%3Fcmd%3Ddisplay_location%26location%3D.1dddc198

The Campus Manager and CiscoView sections seem to be working fine, the part I'm having problems with is RME.

I was finally able to upgrade to LMS 2.6

Version Info:

LMS 2.6

1. CiscoWorks Common Services 3.0.5

2. CiscoView 6.1.5

3. Integration Utility 1.6.0

4. Resource Manager Essentials 4.0.5

5. Campus Manager 4.0.6

Solaris Server

Standalone Config.

I?m still having a few problems as follows:

Device Selector:

Whenever I try to display devices / expand a section, it takes (Not exaggerating) 2-3 minutes to display the list (about 500 devices). This was not an issue before I upgraded to 2.6. Log levels for the device selector are only set to Info, but there are no entries in there since Mar 5 (I upgraded to LMS2.6 on Mar 15). In CiscoView or the campus Manager parts, I can display the entire list with minimal latency. Ideas?

Archive management:

The Daily job that runs fails on all 474 devices. When I view the job, I get this explanation: ?CM0140 Device has been deleted from RME.? The device is in RME (as far as I can tell) and shows up in the device selector. Ideas?

Syslog Analyzer:

Daily backups that are scheduled are not happening. An empty file is created with a filename of something like this: "SyslogPurgedDataBackup1172646007598"

When I try to run a report to see Syslog messages (from a host I know is sending them, I coincidently found a problem with this host by looking through the log file on the CiscoWorks server manually), The report comes back with zero entries. I?ve tried this for all of my devices as well. Other types of reports work fine, like an Inventory report for example. I have tried to Stop / Restart the Syslog Analyzer process thinking maybe that was an issue, but no go. I changed the Log levels for Syslog Analyer and Syslog Analyzer UI to debug.

The SyslogAnalyzerUI.log file is showing many entries like the following:

[ Wed Mar 21 13:49:55 CST 2007 ],DEBUG,[Ajp13Processor[9009][2]],com.cisco.nm.rmeng.util.DCRWrapperAPIs,getDevSelInfo,1153,dcr id: 2670 mdf id : 268438097

[ Wed Mar 21 13:49:55 CST 2007 ],DEBUG,[Ajp13Processor[9009][2]],com.cisco.nm.rmeng.util.DCRWrapperAPIs,getDevSelInfo,1158,Display name : 10.10.17.12

[ Wed Mar 21 13:49:55 CST 2007 ],DEBUG,[Ajp13Processor[9009][2]],com.cisco.nm.rmeng.util.DCRWrapperAPIs,getDevSelInfo,1163,SysObjectID : 1.3.6.1.4.1.9.5.36

It looks like its analyzing the info. The SyslogCollector.log file is full of these entries ONLY:

SyslogCollector - [Thread: SyslogObjectForwarder] INFO , 21 Mar 2007 14:18:10,917, FcssSubscriber - After write

SyslogCollector - [Thread: SyslogObjectForwarder] INFO , 21 Mar 2007 14:18:11,132, FcssSubscriber - After write

SyslogCollector - [Thread: SyslogObjectForwarder] INFO , 21 Mar 2007 14:18:11,355, FcssSubscriber - After write

Ideas why I can?t get a report?

I found this in the RMECSTMServerlog file? Maybe this means something to someone?:

[ Wed Mar 21 12:44:07 CST 2007 ],INFO ,[RMECSTMServer],com.cisco.nm.rmeng.cstm.server.DMgtRunner,run,106,Got daemonmgr message :1111708500

[ Wed Mar 21 13:41:41 CST 2007 ],INFO ,[RMECSTMServer],com.cisco.nm.rmeng.cstm.server.DMgtRunner,run,104,Looping

[ Wed Mar 21 13:41:41 CST 2007 ],INFO ,[RMECSTMServer],com.cisco.nm.rmeng.cstm.server.DMgtRunner,run,106,Got daemonmgr message :1111708500

[ Wed Mar 21 14:11:54 CST 2007 ],INFO ,[RMECSTMServer],com.cisco.nm.rmeng.cstm.server.DMgtRunner,run,104,Looping

[ Wed Mar 21 14:11:54 CST 2007 ],INFO ,[RMECSTMServer],com.cisco.nm.rmeng.cstm.server.DMgtRunner,run,106,Got daemonmgr message :1111708500

Joe Clarke · ‎03-22-2007

This inconsistent data indicates a big problem with the RME database. At this point, you will need to reinitialize your RME database to recover. The command to do that is (with dmgtd shutdown as root):

/opt/CSCOpx/bin/dbRestoreOrig.pl dsn=rmeng dmprefix=RME

Then restart dmgtd. The devices will resync from DCR, and this problem should go away.

View solution in original post

Joe Clarke · ‎03-22-2007

# /etc/init.d/dmgtd stop

# /etc/init.d/dmgtd start

View solution in original post

Joe Clarke · ‎03-21-2007

The device selector issue is a bug. A patch is available at http://www.cisco.com/cgi-bin/Software/Tablebuild/doftp.pl?ftpfile=cisco/cw2000/patches/rme/rme4.0.5-sol-CSCsg518641.tar&app=Tablebuild&status=showC2A that will fix this.

For your config archive problem, you should enable ArchiveMgmt Server debugging under RME > Admin > System Preferences > Loglevel Settings, reproduce the problem, then look at the dcmaservice.log for clues as to why the sync is failing.

For your syslog problems, you will need to enable SyslogAnalyzer debugging under the same location mentioned above as well as SyslogCollector debugging under /opt/CSCOpx/MDC/tomcat/webapps/rme/WEB-INF/classes/com/cisco/nm/rmeng/csc/data/Collector.properties (then restart SyslogCollector).

The logs of interest will be the AnalyzerDebug.log, and the SyslogCollector.log. When debugging the missing messages, you should send one message from a device, and trace it through syslog_info, then look for it in SyslogCollector.log, then look for it in AnalyzerDebug.log. Only when SyslogAnalyzer gets the message will it write to the database.

att0055rj · ‎03-22-2007

I really appreciate your help with this issue. I set the ArchiveMgt Server logging to Debug and let the scheduled job kick off. (Log is attached)

From what I can see, there are basically two different messages that repeat throughout the log:

I really appreciate your help with this issue. I set the ArchiveMgt Server logging to Debug and let the scheduled job kick off. (Log is attached)

From what I can see, there are basically two different messages that repeat throughout the log, the first being:

DEBUG,[Thread-4],com.cisco.nm.rmeng.dcma.configmanager.ConfigManager,addToDeviceIdToReqIdMap,1046,.Device 3058 added to updateResults List.

DEBUG,[Thread-4],com.cisco.nm.rmeng.util.DCRWrapperAPIs,getResultFromQuery,2838,Query: select DCR_ID from DM_Dev_State where NetworkElementID = '3058'

FATAL,[Thread-4],com.cisco.nm.rmeng.util.DCRWrapperAPIs,getDCRId,1084,Device ID: 3058 NOT in RME

DEBUG,[Thread-4],com.cisco.nm.rmeng.dcma.common.DcmaInvClient,getHostName,321,Exception com.cisco.nm.rmeng.util.DeviceNotFoundException: The device: 3058 is not found in Inventory

at com.cisco.nm.rmeng.util.DCRWrapperAPIs.getDCRId(DCRWrapperAPIs.java:1085)

And the second:

DEBUG,[Thread-6],com.cisco.nm.rmeng.inventory.InvAPIs,isValidDeviceID,3450,ResourceBundle set for the logger

DEBUG,[Thread-6],com.cisco.nm.rmeng.inventory.InvAPIs,isValidDeviceID,3458,Querry statement is: select DCR_ID from DM_Dev_State WHERE NetworkElementID = '3031'

DEBUG,[Thread-6],com.cisco.nm.rmeng.util.DCRWrapperAPIs,getResultFromQuery,2838,Query: select DCR_ID from DM_Dev_State WHERE NetworkElementID = '3031'

DEBUG,[Thread-6],com.cisco.nm.rmeng.inventory.InvAPIs,isValidDeviceID,3463,RME does not have this device id: 3031

DEBUG,[Thread-6],com.cisco.nm.rmeng.inventory.InvAPIs,isValidDeviceID,3478,The device id: 3031 is available in RME

ERROR,[Thread-6],com.cisco.nm.rmeng.dcma.configmanager.ConfigManager,performCollection,1543,Config fetch Failed for null

DEBUG,[Thread-6],com.cisco.nm.rmeng.dcma.configmanager.ConfigManager,performCollection,1544,CM0140 Device has been deleted from RME.

CM0204 Could not create DeviceContext for 3031 Cause: Action: Check if required device packages are available in RME.

at com.cisco.nm.rmeng.dcma.configmanager.ConfigManager.getConfigOperator(ConfigManager.java:776)

DEBUG,[Thread-6],com.cisco.nm.rmeng.dcma.configmanager.ConfigManager,storeResults,1180,Device status = 2

DEBUG,[Thread-6],com.cisco.nm.rmeng.dcma.configmanager.ConfigManager,updateFetchResultsForDevice,265,Updating fetch results for 1174532422570

DEBUG,[Thread-6],com.cisco.nm.rmeng.dcma.configmanager.ConfigManager,updateDeviceCount,957,Total devices left = 403

DEBUG,[Thread-6],com.cisco.nm.rmeng.dcma.configmanager.ConfigManager,cleanUpQueueEntries,862,Removed results for device 3031

DEBUG,[Thread-6],com.cisco.nm.rmeng.dcma.configmanager.ConfigManager,updateFetchResultsForDevice,286,Completed device 3031

DEBUG,[Thread-6],com.cisco.nm.rmeng.dcma.cats.CATS,getTransport,127,Getting DeviceContext for DeviceId: 3034

DEBUG,[Thread-6],com.cisco.nm.rmeng.dcma.cats.CATS,getDeviceContext,161,Getting the DeviceContext for DeviceID: 3034

DEBUG,[Thread-6],com.cisco.nm.rmeng.util.DCRWrapperAPIs,getResultFromQuery,2838,Query: select DCR_ID from DM_Dev_State where NetworkElementID = '3034'

FATAL,[Thread-6],com.cisco.nm.rmeng.util.DCRWrapperAPIs,getDCRId,1084,Device ID: 3034 NOT in RME

FATAL,[Thread-6],com.cisco.nm.rmeng.util.rmedaa.RMEDeviceContext,,112,Device with RME ID: 3034 could not be found

FATAL,[Thread-6],com.cisco.nm.rmeng.util.rmedaa.RMEDCFactory,createDC,131,DataRetrivalException occurredcom.cisco.nm.rmeng.util.DataRetrivalException: Error in retrieving data for device: 3034

at com.cisco.nm.rmeng.util.rmedaa.RMEDeviceContext.(RMEDeviceContext.java:113)

Joe Clarke · ‎03-22-2007

This inconsistent data indicates a big problem with the RME database. At this point, you will need to reinitialize your RME database to recover. The command to do that is (with dmgtd shutdown as root):

/opt/CSCOpx/bin/dbRestoreOrig.pl dsn=rmeng dmprefix=RME

Then restart dmgtd. The devices will resync from DCR, and this problem should go away.

att0055rj · ‎03-22-2007

Ok, dumb question.

How would one stop dmgtd, then restart it afterwards? Thanks.

Joe Clarke · ‎03-22-2007

# /etc/init.d/dmgtd stop

# /etc/init.d/dmgtd start

att0055rj · ‎03-21-2007

Update:

If I go through Device Center, the Syslog entires show up in the "24-hour Syslog Message Summary " section:

"

Errors: 433

Warnings: 41

Notifications: 490

Informational: 7

"

The above is a new install that had some glitches haha. Can ya tell?

I am able to click on each section and they will display.

I guess the Syslog Analyzer took a while to catch up again after i restarted it maybe? I can generate a report for it and the messages will show up on the device I mentioned earlier. So i guess mark that one off from the above.