RME 4.0.x: Running configs unavailable

[yjdabear]

RME starts returning "running config unavailable" for a large number (hundreds) of IOS and PIX devices. "Compare Startup vs Running" or "Running vs Latest Archived" errors out:

Startup/Running config fetch failed. Cannot compare configurations.

When choosing "Two versions of the same Device", I can see two generations of running configs for a particular device (#10 and 11), but then I get

HTTP Status 500 - Internal Server Error

--------------------------------------------------------------------------------

type Exception report

message Internal Server Error

description The server encountered an internal error (Internal Server Error) that prevented it from fulfilling this request.

exception

java.lang.NullPointerException

at com.cisco.nm.rmeng.dcma.ui.helper.DCMADiffUIUtil.getConfigletDiff(DCMADiffUIUtil.java:176)

at com.cisco.nm.rmeng.dcma.ui.action.DCMAConfigDiffViewer.perform(DCMAConfigDiffViewer.java:540)

at org.apache.struts.action.ActionServlet.processActionPerform(ActionServlet.java:1786)

at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1585)

at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:491)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:740)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:247)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:193)

at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:256)

at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643)

at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)

at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)

at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)

at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643)

at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)

at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)

at org.apache.catalina.core.StandardContext.invoke(StandardContext.java:2417)

at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:180)

at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643)

at org.apache.catalina.valves.ErrorDispatcherValve.invoke(ErrorDispatcherValve.java:171)

at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641)

at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:172)

at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641)

at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)

at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)

at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:174)

at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643)

at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)

at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)

at org.apache.ajp.tomcat4.Ajp13Processor.process(Ajp13Processor.java:457)

at org.apache.ajp.tomcat4.Ajp13Processor.run(Ajp13Processor.java:576)

at java.lang.Thread.run(Thread.java:534)

--------------------------------------------------------------------------------

Apache Tomcat/4.1.29

All these devices show up under Partially Successful for Config Archive purpose, but other Partially Successful ones have no problem with RME displaying their running configs.

Is there any way to address this situation without reinitializing the RME db?

[Joe Clarke]

It almost looks like someone went and cleaned out your archive on the file system. Enable ArchiveMgmt Client debugging under RME > Admin > System Preferences > Loglevel Settings, and reproduce this error. Then provide the dcmaclient.log.

[yjdabear]

Could you take a look at the dcmaclient.log attached to TAC Case 606284225, when you get a chance? Thanks.

[Joe Clarke]

Looks like you have some corrupt files on the file system. In particular, the config files under /var/adm/CSCOpx/files/rme/dcma/devfiles/532 appear to be corrupt. That appears to be device 10.100.20.112. You might try deleting this device from RME, then re-adding it to see if that corrects the problem you're seeing.

[yjdabear]

This is just one out of hundreds. Does that mean it has to be done against all of them? Any way to prevent what caused the corruption in the first place from recurring?

[Joe Clarke]

The only error in the log points to this one device. However, if other devices exhibit the same problem, then the same thing would apply to them as well. The only prevention against this type of error is to do regular backups.

[yjdabear]

So the cause of this corruption is supposed to be at the file system or hardware level, rather than a bug or defect of RME 4.x? Are the "backups" in your post referring to those produced by LMS 2.6, or by other means such as NetBackup? In the case of the former, how would I restore just the archives for those devices considered corrupt by RME, without setting back other unaffected devices as well? I do have "shadow" directory enabled, if that matters.

[Joe Clarke]

The corruption seen in the logs is most likely not caused by a bug in RME. The files appear to be truncated which is typical of out-of-space conditions or hardware problems.

I refer to an LMS backup. Of course, there is no way to partially restore these backups. In this case, though, since so many devices are affected, a good backup would be the way to go.