Cisco Support Community
cancel
Showing results for 
Search instead for 
Did you mean: 
Announcements

Welcome to Cisco Support Community. We would love to have your feedback.

For an introduction to the new site, click here. And see here for current known issues.

New Member

Cisco Tomcat service hung on CUCM publisher

Hi,

I have CUCM v7.1.5.32900-2.

The web interface became unreachable, restarted Cisco Tomcat service fixed it.

 

Does anyone know why it happened and what can be done to mitigate future occurrence.

 

CUCM Cisco Tomcat Security log:

==============================
2014-06-02 08:09:42,819 ERROR [http-8443-9] impl.DatabaseAccessor - updateLastSuccessfulLoginTime: Failed to execute execute procedure ImsUpdateSuccessfulLoginTime
2014-06-02 08:09:45,691 DEBUG [DefaultQuartzScheduler_Worker-7] impl.AuthenticationDB - Constructor: 
2014-06-02 08:09:51,808 ERROR [http-8443-9] impl.DatabaseAccessor - updateLastSuccessfulLoginTime: Exception Can't load driver java.lang.reflect.InvocationTargetException
2014-06-02 08:09:57,947 ERROR [http-8443-9] impl.AuthenticationDB - authenticateUser: Failed to execute reset last login timeCan't load driver java.lang.reflect.InvocationTargetException
2014-06-02 08:10:43,980 ERROR [http-8443-9] authentication.AuthenticationImpl - loginUtil: SQLException occured. Throwing IMSexception.
java.sql.SQLException: Can't load driver java.lang.reflect.InvocationTargetException
at com.informix.util.IfxErrMsg.getLocSQLException(IfxErrMsg.java:494)
at com.informix.jdbc.IfxDriver.connect(IfxDriver.java:271)
at java.sql.DriverManager.getConnection(DriverManager.java:582)
at java.sql.DriverManager.getConnection(DriverManager.java:207)
at com.cisco.ccm.dbl.Connector.getConn(Connector.java:671)
at com.cisco.ccm.dbl.Connector.connect(Connector.java:588)
at com.cisco.ccm.dbl.Connector.connect(Connector.java:530)
at com.cisco.ccm.dbl.Connector.createTemporaryStatement(Connector.java:851)
at com.cisco.ccm.dbl.Connector.execute(Connector.java:953)
at com.cisco.security.ims.impl.DatabaseAccessor.updateLastLoginTime(DatabaseAccessor.java:943)
at com.cisco.security.ims.impl.AuthenticationDB.authenticateUser(AuthenticationDB.java:253)
at com.cisco.security.ims.authentication.AuthenticationImpl.loginUtil(AuthenticationImpl.java:146)
at com.cisco.security.ims.authentication.AuthenticationImpl.login(AuthenticationImpl.java:74)
at com.cisco.platform.realm.Realm.authenticate(Realm.java:109)
at com.cisco.platform.valve.BasicAuthenticationValve.authenticate(BasicAuthenticationValve.java:219)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:528)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555)
at org.apache.catalina.authenticator.SingleSignOn.invoke(SingleSignOn.java:394)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:857)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at com.informix.jdbc.IfxDriver.connect(IfxDriver.java:254)
... 24 more
Caused by: java.lang.OutOfMemoryError: Java heap space
=============================================
 
Checked Cisco Tomcat Service logs from RTMT, noticed following in Tomcat Manager Logs:
========================================================
Jun 2, 2014 8:00:24 AM org.apache.catalina.core.StandardWrapperValve invoke
SEVERE: Servlet.service() for servlet Manager threw exception
java.lang.OutOfMemoryError: Java heap space
Jun 2, 2014 8:00:27 AM org.apache.catalina.core.StandardWrapperValve invoke
SEVERE: Servlet.service() for servlet Manager threw exception
java.lang.OutOfMemoryError: Java heap space
======================================
5 REPLIES
Cisco Employee

You would need to check the

You would need to check the Event Viewer - System and Application logs on the CUCM publisher and see if there were any alerts as to the memory used by the Cisco Tomcat service. The error reported is OutOfMemory, so the Cisco Tomcat may have hit the threshold of memory that it can take up, and then couldn't function anymore.

Also check if the service created a core using utils core active list.

How many times has this happened in the past? Is this a regular event? If so, this could be a memory leak.

 

Thanks

 

New Member

 Did not find anything

 

Thank you Sreekanth

Did not find anything interesting about memory in System and Application logs.

 

admin:utils core active list

      Size         Date            Core File Name
=================================================================
 299252 KB   2014-04-18 19:39:50   core.28338.11.cef.1397864390
 118156 KB   2012-11-17 02:00:36   core.13997.6.cimlistener.1353135636
 118284 KB   2011-10-29 03:04:30   core.14005.11.cimlistener.1319871870
 603908 KB   2013-09-07 02:07:57   core.18107.11.cef.1378534077
 368192 KB   2014-02-03 05:14:23   core.25216.11.cef.1391422462

Cisco Employee

Ok, what is the RTMT version

Ok, what is the RTMT version that is being used to log in to the servers? Is it the client that has been downloaded from this CUCM versions's Plugins page? There is a defect that a lot of customers hit where catalina would have OutOfMemory errors and Tomcat would stop responding due to the large number of expired RTMT requests stuck in memory. CSCtj50884 is the defect.

 

Also, please log in to the platform using SSH and run the following command and check if you see a .hprof file.

file list activelog tomcat/logs/*

New Member

 RTMT 8.1, downloaded from

 

RTMT 8.1, downloaded from same CUCM plug-ins

 

Yes, there are two:

heap.17222.tomcat.1391005862.hprof
heap.29284.tomcat.1401708513.hprof

 

Cisco Employee

Can you upload them here?

Can you upload them here? Will need to decode those heaps and check for suspects.

416
Views
0
Helpful
5
Replies
CreatePlease login to create content