LMS Database Troubleshooting

Document

Aug 7, 2011 2:15 PM
Mar 10th, 2010

Introduction

This document applies to CiscoWorks LAN Management Solution 3.1 through Cisco Prime LAN Management Solution 4.1.

LAN Management Solution uses Sybase-like embedded databases from SQL Anywhere.  Each of the main applications in LMS contains at least one database.  Assuming all of the LMS is installed (including the Health and Utilization Monitor add-on), a server will have the following database engine processes.  NOTE: LMS 4.x will have all of these databases installed except where noted:

Daemon ProcessLMS Application (Device Management Function)
CmfDbEngineCommon Services
RMEDbEngineResource Manager Essentials (Inventory, Config and Image Management)
ANIDbEngineCampus Manager (Network Topology, Layer 2 Services and User Tracking)
IpmDbEngineInternetwork Performance Monitor (IPSLA Performance Management)
INVDbEngineDevice Fault Manager (Fault Management)
EPMDbEngineDevice Fault Manager (Fault Management)
FHDbEngineDevice Fault Manager (Fault Management)
OpsxmlDbEngineCiscoWorks Assistant (LMS 3.x only)
UPMDbEngineHealth and Utilization Monitor (Device Performance Management)

Changing the Database Password

By default, the LMS installer will set all of the databases password to a random value.  If there is a need to access the database directly (for troubleshooting purposes), or if you notice errors in the logs pertaining to database login failures, you made need to change the database password.  These login failures typically look like:

java.sql.SQLException: JZ00L: Login failed.

Examine the SQLWarnings chained to this exception for the reason(s).

To change the database password for any of the SQL Anywhere databases in LMS use the dbpasswd.pl command.  This command is found in the NMSROOT/bin directory.  NMSROOT is the path into which LMS was installed.  By default, this is C:\PROGRA~1\CSCOpx on Windows and /opt/CSCOpx on Solaris and the LMS appliance (LMS 4.1 only).  You can either change the password on all databases, or change the password on a specific database.  To change the password on all databases, use the following command:

Solaris and the LMS appliance:
/opt/CSCOpx/bin/dbpasswd.pl all

Windows:
NMSROOT\bin\perl.exe NMSROOT\bin\dbpasswd.pl all

NOTE: On the LMS appliance (LMS 4.1 only), you must enter the root shell mode in order to perform any of these commands.  To enter the root shell, connect to the appliance as the sysadmin user (this username can be changed at installation time), then enter the command shell.  When prompted, enter the root shell password password.

NOTE: Daemon Manager must be shutdown prior to changing the database passwords.  To shut down Daemon Manager, run the following command:

Solaris and the LMS appliance:
/etc/init.d/dmgtd stop

Windows:
net stop crmdmgtd

To change the password of individual databases, you must specify the datasource name (DSN).  You can use the listdsn argument to dbpasswd.pl to get a list of all DSNs.  To save you some time, the following table lists the DSN for each of the databases in LMS.

DatabaseDatasource Name
CmfDbEnginecmf
RMEDbEnginermeng
ANIDbEngineani
IpmDbEngineipm
INVDbEnginedfmInv
EPMDbEnginedfmEpm
FHDbEnginedfmFh
OpsxmlDbEngineopsxml (LMS 3.x only)
UPMDbEngineupm

Run the dbpasswd.pl command with the dsn argument to change an individual database password.  For example, to change the RME (Inventory, Config and Software Image Management) database password, run the following command:

Solaris and the LMS appliance:
/opt/CSCOpx/bin/dbpasswd.pl dsn=rmeng

Windows:
NMSROOT\bin\perl.exe NMSROOT\bin\dbpasswd.pl dsn=rmeng

Recovering From a Server Crash

If an LMS server crashes or is not properly powered off, a database's transaction log may become corrupted.  It is certainly also possible for the database itself to be corrupt; however, since LMS uses transactions, there is a bigger chance that only the transaction log will be corrupted. Typically, it is apparent that transaction log recovery needs to be done if the pdshow command reports that the database engine is in a "Never started" or "Failed to start" state (although the latter could indicate more serious database corruption).

To recover when a database transaction log becomes corrupted, first shutdown Daemon Manager.  See the steps in the section on changing the database password on how to shutdown Daemon Manager.

Once Daemon Manager is shutdown, delete the database's transaction log if it still exists.  The transaction logs will be found in the database directory.  This is NMSROOT/databases/DSN where DSN is the database datasource name.  See the table in the section on changing the database password to get a list of datasource names.  For example, the CMFDEngine (Common Services database) transaction log will be in NMSROOT/databases/cmf.  The log file will be named DSN.log (e.g. cmf.log).  These log files are binary, and should not be opened.

Once the database transaction log has been deleted, the database must be forced closed (i.e. indicating that there are no more pending transactions).  This is done using the dbsrv10 command.  For example, to force the cmf database closed (thus allowing it to start properly), use the following command:

Solaris:
env LD_LIBRARY_PATH=/opt/CSCOpx/objects/db/lib64 /opt/CSCOpx/objects/db/bin64/dbsrv10 -f /opt/CSCOpx/databases/cmf/cmf.db

LMS appliance:

env LD_LIBRARY_PATH=/opt/CSCOpx/objects/db/lib64 /opt/CSCOpx/objects/db/bin64s/dbsrv10 -f /opt/CSCOpx/databases/cmf/cmf.db
Windows:
NMSROOT\objects\db\win32\dbsrv10.exe -f NMSROOT\databases\cmf\cmf.db

Reinitializing a Database

It may be that a database file itself becomes corrupted.  The can happen after a server crash, or if a file system runs out of disk space (or a variety of other causes).  There is no one symptom of this.  The most obvious symptom is that pdshow reports the database as "Failed to start" (though in rare cases, this could be a symptom of transaction log corruption).  You may also note assertion errors in the /var/adm/CSCOpx/log/daemons.log (on Solaris and the LMS appliance) or NMSROOT\log\syslog.log (on Windows)  files.  You can also check for most types of corruption by running validation on the database DSN.  For example, to validate the cmf database, run the following command:

Solaris and the LMS appliance:

/opt/CSCOpx/objects/db/conf/configureDb.pl action=validate dsn=cmf

Windows:

NMSROOT\bin\perl.exe NMSROOT\objects\db\conf\configureDb.pl action=validate dsn=cmf

Reinitializing a database will destroy ALL data associated with that database.  If you need to reinitialize the cmf database, you will lose ALL data in ALL LMS applications.  Therefore, it is highly recommended that you perform regular CiscoWorks data backups so that you can restore your data if you encounter database corruption.

Below are the commands to reinitialize each of the databases.  If you reinitialize multiple databases, you must do so in the right order.  The list below is in the correct order.  If you reinitialize the cmf database, you should reinitialize ALL databases.  Before reinitializing the database, you must shutdown Daemon Manager.  See the instructions above in the section on changing the database passwords on how to shutdown Daemon Manager.

Solaris and the LMS appliance:

/opt/CSCOpx/bin/dbRestoreOrig.pl dsn=cmf dmprefix=Cmf

/opt/CSCOpx/bin/dbRestoreOrig.pl dsn=rmeng dmprefix=RME

/opt/CSCOpx/bin/dbRestoreOrig.pl dsn=ani dmprefix=ANI

/opt/CSCOpx/bin/dbRestoreOrig.pl dsn=ipm dmprefix=Ipm

/opt/CSCOpx/bin/dbRestoreOrig.pl dsn=dfmInv dmprefix=INV

/opt/CSCOpx/bin/dbRestoreOrig.pl dsn=dfmEpm dmprefix=EPM

/opt/CSCOpx/bin/dbRestoreOrig.pl dsn=dfmFh dmprefix=FH

/opt/CSCOpx/bin/dbRestoreOrig.pl dsn=opsxml dmprefix=Opsxml

(note: the previous command is only required on LMS 3.x)

/opt/CSCOpx/bin/dbRestoreOrig.pl dsn=upm dmprefix=UPM

Windows:

NMSROOT\bin\perl.exe NMSROOT\bin\dbRestoreOrig.pl dsn=cmf dmprefix=Cmf

NMSROOT\bin\perl.exe NMSROOT\bin\dbRestoreOrig.pl dsn=rmeng dmprefix=RME

NMSROOT\bin\perl.exe NMSROOT\bin\dbRestoreOrig.pl dsn=ani dmprefix=ANI

NMSROOT\bin\perl.exe NMSROOT\bin\dbRestoreOrig.pl dsn=ipm dmprefix=Ipm

NMSROOT\bin\perl.exe NMSROOT\bin\dbRestoreOrig.pl dsn=dfmInv dmprefix=INV

NMSROOT\bin\perl.exe NMSROOT\bin\dbRestoreOrig.pl dsn=dfmEpm dmprefix=EPM

NMSROOT\bin\perl.exe NMSROOT\bin\dbRestoreOrig.pl dsn=dfmFh dmprefix=FH

NMSROOT\bin\perl.exe NMSROOT\bin\dbRestoreOrig.pl dsn=opsxml dmprefix=Opsxml

(note: the previous command is only required on LMS 3.x)

NMSROOT\bin\perl.exe NMSROOT\bin\dbRestoreOrig.pl dsn=upm dmprefix=UPM

Note: All three of the DFM (Fault Management) databases (dfmInv, dfmEpm, and dfmFh) should be reinitialized at the same time.  DFM (Fault Management) also uses another datastore which must also be deleted.  After reinitializing the DFM SQL Anywhere databases, delete the DFM.rps and DFM1.rps files under NMSROOT/objects/smarts/local/repos/icf.

Actions

This Document

Related Content