Cisco Support Community
cancel
Showing results for 
Search instead for 
Did you mean: 

CUCM Tomcat memory leak requires RTMT client upgrade to fix - CSCtj50884

Description

CSCtj50884 covers a leak that will result in the Tomcat web service running out of memory.  This will disrupt web services which includes the CCMAdmin and CCMUser web sites, Extension Mobility operations, and even IPCC agent logins.

Getting the fix

There are fixes available for every major version of CUCM impacted by this defect.

From cisco.com

6.1(5)SU3

7.1(5)SU4   this image will be posted soon to cisco.com

8.0(3a)SU2

An SU for 8.5(1) is planned but no release date is available.

From TAC via engineering special

Since all CUCM ES patches are cumulative if you are running an ES later than the ones listed below you have the fix.

6.1.5.13039-1

7.1.5.33043-1

8.0.3.22026-1

8.5.1.12008-3

Applying the fix

After upgrading CUCM to a version with the fix the last step is to download the new RTMT client from the upgraded server and completely reinstall  RTMT on every PC that connects to CUCM.  Failure to install the new RTMT client will expose the server to more failures even after applying the upgrade.

Symptoms

Tomcat memory leaks will show up over time and will range from slow response to web service through intermittent failures for some web services and eventually no web services will work.

You can recover service immediately by restarting Tomcat from the CLI with the command utils service restart Cisco Tomcat

In the Cisco Tomcat catalina.out log file you will see errors such as this:

Dumping heap to /usr/local/thirdparty/jakarta-tomcat/logs/java_pid5555.hprof ...
Heap dump file created [598225933 bytes in 69.829 secs]
Jan 01, 2011 12:14:55 AM org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor processChildren
SEVERE: Exception invoking periodic operation:
java.lang.OutOfMemoryError: Java heap space
     at java.lang.StringCoding$CharsetSE.encode(StringCoding.java:334)
     at java.lang.StringCoding.encode(StringCoding.java:378)
     at java.lang.String.getBytes(String.java:812)
     at java.io.UnixFileSystem.getLastModifiedTime(Native Method)
     at java.io.File.lastModified(File.java:795)

This particular defect is not the only way that Tomcat can run out of heap memory and throw this error.  The .hprof file is generated by the system any time the heap space is exhaused and its contents are the only way to prove conclusively the cause of the fault.  The details for CSCtj50884 in Bug Toolkit contains instructions for decoding the .hprof file if you would like to get 100% confirmation.

Version history
Revision #:
1 of 1
Last update:
‎05-03-2011 02:02 PM
Updated by:
 
Labels (1)
Comments
New Member

Hi,

I suspect the V8.6.2.21900-5 to be also affected by this bug, and I hope there is a solution on the road.

After a cluster reboot on anew implemented BE 6000, because of some TFTP file uploads, the WEb SErvices (Tomcat, Axl,...) did not coming up anymore on the Pub and on the sub....Strange...,and did show "

Commanded Out of Service" and "Starting" (Tomcat)

None of my troubleshooting trials did help.  Now I think about an Upgrade with the same Version on the second partition to force a reinstallation of the web services.

admin:utils service list

Requesting service status, please wait...

System SSH [STARTED]

Cluster Manager [STARTED]

Service Manager is running

Getting list of all services

>> Return code = 0

A Cisco DB[STARTED]

A Cisco DB Replicator[STARTED]

Cisco AMC Service[STARTED]

Cisco AXL Web Service[STOPPED]  Commanded Out of Service

Cisco Audit Event Service[STARTED]

Cisco Bulk Provisioning Service[STARTED]

Cisco CAR DB[STARTED]

Cisco CAR Scheduler[STARTED]

Cisco CAR Web Service[STOPPED]  Commanded Out of Service

Cisco CDP[STARTED]

Cisco CDP Agent[STARTING]

Cisco CDR Agent[STARTED]

Cisco CDR Repository Manager[STARTED]

Cisco CTIManager[STARTED]

Cisco CTL Provider[STARTED]

Cisco CallManager[STARTED]

Cisco CallManager Admin[STOPPED]  Commanded Out of Service

Cisco CallManager Cisco IP Phone Services[STOPPED]  Commanded Out of Service

Cisco CallManager Personal Directory[STOPPED]  Commanded Out of Service

Cisco CallManager SNMP Service[STARTED]

Cisco CallManager Serviceability[STOPPED]  Commanded Out of Service

Cisco CallManager Serviceability RTMT[STOPPED]  Commanded Out of Service

Cisco Certificate Authority Proxy Function[STARTED]

Cisco Certificate Change Notification[STARTED]

Cisco Certificate Expiry Monitor[STARTED]

Cisco Change Credential Application[STOPPED]  Commanded Out of Service

Cisco DHCP Monitor Service[STARTED]

Cisco DRF Local[STARTED]

Cisco DRF Master[STARTED]

Cisco Database Layer Monitor[STARTED]

Cisco Dialed Number Analyzer[STOPPED]  Commanded Out of Service

Cisco Dialed Number Analyzer Server[STARTED]

Cisco DirSync[STARTED]

Cisco Extended Functions[STARTED]

Cisco Extension Mobility[STOPPED]  Commanded Out of Service

Cisco Extension Mobility Application[STOPPED]  Commanded Out of Service

Cisco IP Manager Assistant[STOPPED]  Commanded Out of Service

Cisco IP Voice Media Streaming App[STARTED]

Cisco License Manager[STARTED]

Cisco Log Partition Monitoring Tool[STARTED]

Cisco RIS Data Collector[STARTED]

Cisco RTMT Reporter Servlet[STOPPED]  Commanded Out of Service

Cisco SOAP - CDRonDemand Service[STARTED]

Cisco SOAP - CallRecord Service[STARTED]

Cisco Serviceability Reporter[STARTED]

Cisco Syslog Agent[STARTING]

Cisco TAPS Service[STARTED]

Cisco Tftp[STARTED]

Cisco Tomcat[STARTING]

Cisco Tomcat Stats Servlet[STOPPED]  Commanded Out of Service

Cisco Trace Collection Service[STARTED]

Cisco Trace Collection Servlet[STOPPED]  Commanded Out of Service

Cisco Trust Verification Service[STARTED]

Cisco UXL Web Service[STOPPED]  Commanded Out of Service

Cisco Unified Mobile Voice Access Service[STOPPED]  Commanded Out of Service

Cisco User Data Services[STOPPED]  Commanded Out of Service

Cisco WebDialer Web Service[STOPPED]  Commanded Out of Service

Host Resources Agent[STARTED]

MIB2 Agent[STARTED]

Platform SOAP Services[STOPPED]  Commanded Out of Service

SNMP Master Agent[STARTED]

SOAP - Diagnostic Portal Database Service[STARTED]

SOAP -Log Collection APIs[STARTED]

SOAP -Performance Monitoring APIs[STARTED]

SOAP -Real-Time Service APIs[STARTED]

System Application Agent[STARTED]

Cisco Messaging Interface[STOPPED]  Service Not Activated

Primary Node =true

Any quick help idea ?

None of simple starts or restarts did help. I've also have a look into the tomcat cert to check the device name.

Jacky

New Member

We are having very slow response from our CUCM web interface. (ver 8.6.2.21900-5)  A restart of the Tomcat service seems to help, but a more permanant fix is needed.

Did you ever figure out what was causing it, or find a patch?