Cisco Support Community
cancel
Showing results for 
Search instead for 
Did you mean: 

RTMT FOR CUCM 7.1.X

Hi

We are experiencing an issue with the RTMT

Tomcat - java.lang.OutOfMemoryError due to large number of RTMT requests

So we need to disable the RTMT for the specific server

we have 3 servers in a cluster

the correct Procedure is to stop the Cisco CallManager Serviceability RTMT service for the specific server??

Regards

cc

Please rate all useful posts Regards Chrysostomos ""The Most Successful People Are Those Who Are Good At Plan B""
Everyone's tags (3)
1 ACCEPTED SOLUTION

Accepted Solutions
Hall of Fame Super Red

Re: RTMT FOR CUCM 7.1.X

Hi CC,

If you have an issue where all the services are down and you can't

access CUCM via gui or cli then I would suggest that you may be seeing

a different issue altogether than the one listed in the bug notes.

I would go ahead and open a TAC case here

You can open the case right from this thread. Have a look.

Announcement: Create Service Request from CSC Discussions!

We are pleased to announce the ability to create Service Requests from CSC discussion threads. Click here to learn more and provide feedback.

Cheers!

Rob

"Always movin' ahead and never lookin' back" - Springsteen

7 REPLIES
Hall of Fame Super Red

RTMT FOR CUCM 7.1.X

Hi CC,

It looks like you are hitting this bug;

CSCtj50884 - Tomcat - java.lang.OutOfMemoryError due to large number of RTMT requests

Description

Symptom:

Symptoms include:
- Slow response on access to UCM web pages.
- Failed Extension Mobility operations due to timeout or unreachable server.
- Failed IPCC agent logins due to failing AXL requests.
- java.lang.OutOfMemoryError in Tomcat logs (catalina.out).
Conditions:

UCM 6.1.4, 6.1.5, 7.1.5, 8.0.3, 8.5.1.
This issue can be seen even if older versions of the RTMT client are used to
connect to a server with the fix.
To mitigate this you must download and install the newer RTMT client
from CUCM after upgrading to a version with the fix.
Note: If RTMT clients, "old" clients, are actively attempting to connect
when the UCM nodes come online after an upgrade the memory leak WILL return. It
is recommended to stop all RTMT clients before the upgrade is executed, Tomcat
access logs will show the IP of any connected clients. After the Publisher is
upgraded the updated RTMT client may be used to monitor the upgrade of the
remaining nodes.

Workaround:
Restart Tomcat from the CLI of the server with the command utils service
restart Cisco Tomcat

Additional Info:
To determine if a reboot is needed:
To avoid unexpected web service outages, monitor the Cisco Tomcat JVM counters
with RTMT or the CLI and proactively restart Tomcat if the TOTAL heap size =
MAX heap size. Tomcat will continue to run when TOTAL = MAX while FREE is > 0.

To verify a server is experiencing CSCtj50884:
1.) Download the Tomcat Java heap file (*.hprof) automatically generated on the
CUCM server to your laptop or desktop
file get activelog tomcat/logs/*
2.) Run the following Java command against the downloaded heap:
jhat -J-mx2048m .hprof
It will take a few minutes and eventually start a HTTP server on port 7000.
Note, jhat is provided as part of the Java Development Kit (Standard Edition). 
You will need to install the Sun/Oracle JDK locally in order to run the jhat
command.
http://www.oracle.com/technetwork/java/javase/downloads/index.html
3.) Access the server via any web browser:
http://:7000
4.) Select the "Execute Object Query Language (OQL) query" option at the very
bottom of the page.
5.) Enter the following OQL query and click the Execute button.
select count(heap.objects('org.apache.catalina.session.StandardSession'),
"it.isValid.toString()=='false'")
Crashes due to CSCtj50884 will have thousands and thousands of invalid
sessions.
Additional Info #2:
Eclipse Memory Analyzer (MAT) may also be used to confirm this DDTS,
http://www.eclipse.org/mat/.
Find major memory user:
1) Download & install MAT, http://www.eclipse.org/mat/
2) Open heap dump in MAT
3) Select "Leak Suspects Report" & "Finish"
Find number of outstanding connections:
1) Open the Open Object Query Language studio
2) Execute this query,
SELECT o.id.toString(), o.isValid, o.creationTime FROM
org.apache.catalina.session.StandardSession o
WHERE (o.isValid=false)
If the major user is "org.apache.catalina.authenticator.SingleSignOn" & the
number of outstanding connections (total entries) are in the 1000's or 10's of
1000's CSCtj50884 is root cause of the failure.

Details

First Found in:                          (5)

6.1(5),8.0(3),8.5(1),7.1,8.6(0.96071.5)

Status:

Fixed

Last Modified:

Aug 16,2012

Fixed in:                          (20)

9.0(0.96000.1),9.0(0.95010.1),8.6(1.98000.43)

8.6(1.95020.80),8.6(1.10001.1),8.6(1.10000.43)

8.6(1.10000.1),8.6(0.99981.2),8.6(0.99081.6)

8.6(0.99081.4),8.6(0.98000.8),8.6(0.98000.24)

8.6(0.95180.9),8.5(1.12008.3),8.0(3.22900.2)

8.0(3.22026.1),7.1(5.33043.1),7.0(2.24055.1)

6.1(5.13900.4),6.1(5.13039.1)

Less

Product:

Cisco Unified Communications Manager (CallManager)

Platform:

Dependent

Severity:

2 - severe

Cheers!

Rob

"Always movin' ahead and never lookin' back" - Springsteen

RTMT FOR CUCM 7.1.X

Hi Rob

Long time ago

excellent post , but what about the workaround

if we disable the RTMT (service ) in the specific cucm it will be ok?

if we disable this service?

Cisco CallManager Serviceability RTMT

Please rate all useful posts Regards Chrysostomos ""The Most Successful People Are Those Who Are Good At Plan B""
Hall of Fame Super Red

RTMT FOR CUCM 7.1.X

Hi CC,

I would go with the first note in the "workaround"

Restart Tomcat from the CLI of the server with the command utils service
restart Cisco Tomcat

Then I would try to not keep RTMT nailed up all the time. If that doesn't work

then reboot the CUCM node/servers as I'm not sure that you can stop the

service you referenced (auto-start).

Cheers!

Rob

"Always movin' ahead and never lookin' back" - Springsteen

RTMT FOR CUCM 7.1.X

Hi

When this issue coming the server is up but all the services are down

we cant access the CUCM either from gui or cli

you mean that i cant disable the RTMT in general??

note that teh RTMT is not open..But is configured and it send an email alert for any issue

Please rate all useful posts Regards Chrysostomos ""The Most Successful People Are Those Who Are Good At Plan B""
Hall of Fame Super Red

Re: RTMT FOR CUCM 7.1.X

Hi CC,

If you have an issue where all the services are down and you can't

access CUCM via gui or cli then I would suggest that you may be seeing

a different issue altogether than the one listed in the bug notes.

I would go ahead and open a TAC case here

You can open the case right from this thread. Have a look.

Announcement: Create Service Request from CSC Discussions!

We are pleased to announce the ability to create Service Requests from CSC discussion threads. Click here to learn more and provide feedback.

Cheers!

Rob

"Always movin' ahead and never lookin' back" - Springsteen

RTMT FOR CUCM 7.1.X

thank you Rob for the infos exchange

i will give a rate for the all the answers that you  gave

Thnak you

chrysostomos

Please rate all useful posts Regards Chrysostomos ""The Most Successful People Are Those Who Are Good At Plan B""
Hall of Fame Super Red

RTMT FOR CUCM 7.1.X

Hi Chrysostomos,

You are always welcome my friend Sorry we weren't able

to come up with a solution here. Please let us know how this works out.

Cheers!

Rob

"Always movin' ahead and never lookin' back" - Springsteen

669
Views
0
Helpful
7
Replies
CreatePlease to create content