Hi CC,
It looks like you are hitting this bug;
CSCtj50884 - Tomcat - java.lang.OutOfMemoryError due to large number of RTMT requests
Description
Symptom:
Symptoms include:
- Slow response on access to UCM web pages.
- Failed Extension Mobility operations due to timeout or unreachable server.
- Failed IPCC agent logins due to failing AXL requests.
- java.lang.OutOfMemoryError in Tomcat logs (catalina.out).
Conditions:
UCM 6.1.4, 6.1.5, 7.1.5, 8.0.3, 8.5.1.
This issue can be seen even if older versions of the RTMT client are used to
connect to a server with the fix.
To mitigate this you must download and install the newer RTMT client
from CUCM after upgrading to a version with the fix.
Note: If RTMT clients, "old" clients, are actively attempting to connect
when the UCM nodes come online after an upgrade the memory leak WILL return. It
is recommended to stop all RTMT clients before the upgrade is executed, Tomcat
access logs will show the IP of any connected clients. After the Publisher is
upgraded the updated RTMT client may be used to monitor the upgrade of the
remaining nodes.
Workaround:
Restart Tomcat from the CLI of the server with the command utils service
restart Cisco Tomcat
Additional Info:
To determine if a reboot is needed:
To avoid unexpected web service outages, monitor the Cisco Tomcat JVM counters
with RTMT or the CLI and proactively restart Tomcat if the TOTAL heap size =
MAX heap size. Tomcat will continue to run when TOTAL = MAX while FREE is > 0.
To verify a server is experiencing CSCtj50884:
1.) Download the Tomcat Java heap file (*.hprof) automatically generated on the
CUCM server to your laptop or desktop
file get activelog tomcat/logs/*
2.) Run the following Java command against the downloaded heap:
jhat -J-mx2048m .hprof
It will take a few minutes and eventually start a HTTP server on port 7000.
Note, jhat is provided as part of the Java Development Kit (Standard Edition).
You will need to install the Sun/Oracle JDK locally in order to run the jhat
command.
http://www.oracle.com/technetwork/java/javase/downloads/index.html
3.) Access the server via any web browser:
http://:7000
4.) Select the "Execute Object Query Language (OQL) query" option at the very
bottom of the page.
5.) Enter the following OQL query and click the Execute button.
select count(heap.objects('org.apache.catalina.session.StandardSession'),
"it.isValid.toString()=='false'")
Crashes due to CSCtj50884 will have thousands and thousands of invalid
sessions.
Additional Info #2:
Eclipse Memory Analyzer (MAT) may also be used to confirm this DDTS,
http://www.eclipse.org/mat/.
Find major memory user:
1) Download & install MAT, http://www.eclipse.org/mat/
2) Open heap dump in MAT
3) Select "Leak Suspects Report" & "Finish"
Find number of outstanding connections:
1) Open the Open Object Query Language studio
2) Execute this query,
SELECT o.id.toString(), o.isValid, o.creationTime FROM
org.apache.catalina.session.StandardSession o
WHERE (o.isValid=false)
If the major user is "org.apache.catalina.authenticator.SingleSignOn" & the
number of outstanding connections (total entries) are in the 1000's or 10's of
1000's CSCtj50884 is root cause of the failure.
Details
First Found in: (5)
6.1(5),8.0(3),8.5(1),7.1,8.6(0.96071.5)
Last Modified:
Aug 16,2012
Fixed in: (20)
9.0(0.96000.1),9.0(0.95010.1),8.6(1.98000.43)
8.6(1.95020.80),8.6(1.10001.1),8.6(1.10000.43)
8.6(1.10000.1),8.6(0.99981.2),8.6(0.99081.6)
8.6(0.99081.4),8.6(0.98000.8),8.6(0.98000.24)
8.6(0.95180.9),8.5(1.12008.3),8.0(3.22900.2)
8.0(3.22026.1),7.1(5.33043.1),7.0(2.24055.1)
6.1(5.13900.4),6.1(5.13039.1)
Product:
Cisco Unified Communications Manager (CallManager)
Cheers!
Rob
"Always movin' ahead and never lookin' back" - Springsteen
