Cisco Support Community
cancel
Showing results for 
Search instead for 
Did you mean: 
Announcements

Welcome to Cisco Support Community. We would love to have your feedback.

For an introduction to the new site, click here. And see here for current known issues.

CUCM 9.1.1a CTIManager service core dump

Has anyone seen the CTIManager service core dump in 9.1.1a yet?  Not finding much in bug toolkit.

CUCM 9.1.1.20000-5

Core File Analysis, only thing here is "SDL Router Services declared dead. This may be due to high CPU usage or blocked function. Attempting to restart " at the end of the analysis. This is on a VM (SAN Attached) and another call manager cluster did not have any issues.

Reading symbols from /usr/local/cm/bin/CTIManager...done.

[New Thread 3448]

[New Thread 4015]

[New Thread 3509]

[New Thread 3505]

[New Thread 3450]

[New Thread 3449]

[New Thread 3447]

[New Thread 3446]

[New Thread 3445]

[New Thread 3066]

[New Thread 3064]

[New Thread 3063]

[New Thread 3062]

[New Thread 3061]

[New Thread 3060]

[New Thread 3059]

[New Thread 12880]

[New Thread 12879]

[New Thread 7432]

[New Thread 7431]

[New Thread 7342]

[New Thread 7211]

[New Thread 7186]

[New Thread 7145]

[New Thread 6681]

Reading symbols from /usr/local/cm/lib/libJNIPIWriter.so...done.

Loaded symbols for /usr/local/cm/lib/libJNIPIWriter.so

Reading symbols from /usr/local/cm/lib/libssl.so.0.9.8...(no debugging symbols found)...done.

Loaded symbols for /usr/local/cm/lib/libssl.so.0.9.8

Reading symbols from /usr/local/cm/lib/libpcre.so.0...done.

Loaded symbols for /usr/local/cm/lib/libpcre.so.0

Reading symbols from /usr/local/cm/lib/libdbnotify.so...done.

Loaded symbols for /usr/local/cm/lib/libdbnotify.so

Reading symbols from /usr/local/cm/lib/libldap-2.4.so.2...done.

Loaded symbols for /usr/local/cm/lib/libldap-2.4.so.2

Reading symbols from /usr/local/cm/lib/liblber-2.4.so.2...done.

Loaded symbols for /usr/local/cm/lib/liblber-2.4.so.2

Reading symbols from /usr/local/cm/lib/libdbl.so...done.

Loaded symbols for /usr/local/cm/lib/libdbl.so

Reading symbols from /usr/local/cm/lib/libpreferences.so...done.

Loaded symbols for /usr/local/cm/lib/libpreferences.so

Reading symbols from /usr/local/cm/lib/libodbc++-mt.so.4...done.

Loaded symbols for /usr/local/cm/lib/libodbc++-mt.so.4

Reading symbols from /usr/local/cm/db/informix/lib/cli/libifdmr.so...(no debugging symbols found)...done.

Loaded symbols for /usr/local/cm/db/informix/lib/cli/libifdmr.so

Reading symbols from /usr/local/cm/db/informix/lib/cli/libthcli.so...(no debugging symbols found)...done.

Loaded symbols for /usr/local/cm/db/informix/lib/cli/libthcli.so

Reading symbols from /usr/local/cm/db/informix/lib/esql/libifgls.so...(no debugging symbols found)...done.

Loaded symbols for /usr/local/cm/db/informix/lib/esql/libifgls.so

Reading symbols from /usr/local/cm/lib/libCryptoUtil.so...done.

Loaded symbols for /usr/local/cm/lib/libCryptoUtil.so

Reading symbols from /usr/local/cm/lib/libcrypto.so.0.9.8...(no debugging symbols found)...done.

Loaded symbols for /usr/local/cm/lib/libcrypto.so.0.9.8

Reading symbols from /usr/local/cm/lib/libxerces-c.so.27...(no debugging symbols found)...done.

Loaded symbols for /usr/local/cm/lib/libxerces-c.so.27

Reading symbols from /usr/local/platform/lib/libACE.so.5.5.1...(no debugging symbols found)...done.

Loaded symbols for /usr/local/platform/lib/libACE.so.5.5.1

Reading symbols from /usr/local/cm/lib/libstlport.so.5.1...(no debugging symbols found)...done.

Loaded symbols for /usr/local/cm/lib/libstlport.so.5.1

Reading symbols from /usr/lib/libz.so.1...(no debugging symbols found)...done.

Loaded symbols for /usr/lib/libz.so.1

Reading symbols from /lib/libdl.so.2...(no debugging symbols found)...done.

Loaded symbols for /lib/libdl.so.2

Reading symbols from /lib/libcap.so.1...(no debugging symbols found)...done.

Loaded symbols for /lib/libcap.so.1

Reading symbols from /lib/libgcc_s.so.1...(no debugging symbols found)...done.

Loaded symbols for /lib/libgcc_s.so.1

Reading symbols from /lib/libc.so.6...(no debugging symbols found)...done.

Loaded symbols for /lib/libc.so.6

Reading symbols from /lib/libresolv.so.2...(no debugging symbols found)...done.

Loaded symbols for /lib/libresolv.so.2

Reading symbols from /lib/libpthread.so.0...(no debugging symbols found)...done.

[Thread debugging using libthread_db enabled]

Loaded symbols for /lib/libpthread.so.0

Reading symbols from /usr/local/cm/lib/libuuidpp.so...done.

Loaded symbols for /usr/local/cm/lib/libuuidpp.so

Reading symbols from /lib/libm.so.6...(no debugging symbols found)...done.

Loaded symbols for /lib/libm.so.6

Reading symbols from /lib/librt.so.1...(no debugging symbols found)...done.

Loaded symbols for /lib/librt.so.1

Reading symbols from /usr/local/cm/db/informix/lib/esql/libifglx.so...(no debugging symbols found)...done.

Loaded symbols for /usr/local/cm/db/informix/lib/esql/libifglx.so

Reading symbols from /lib/libcrypt.so.1...(no debugging symbols found)...done.

Loaded symbols for /lib/libcrypt.so.1

Reading symbols from /usr/lib/libstdc++.so.6...(no debugging symbols found)...done.

Loaded symbols for /usr/lib/libstdc++.so.6

Reading symbols from /lib/ld-linux.so.2...(no debugging symbols found)...done.

Loaded symbols for /lib/ld-linux.so.2

Reading symbols from /lib/libnss_files.so.2...(no debugging symbols found)...done.

Loaded symbols for /lib/libnss_files.so.2

Reading symbols from /usr/local/cm/db/informix/lib/esql/igo4a304.so...(no debugging symbols found)...done.

Loaded symbols for /usr/local/cm/db/informix/lib/esql/igo4a304.so

Reading symbols from /lib/libnss_dns.so.2...(no debugging symbols found)...done.

Loaded symbols for /lib/libnss_dns.so.2

Core was generated by `/usr/local/cm/bin/CTIManager'.

Program terminated with signal 6, Aborted.

#0  0x08e8e206 in raise () from /lib/libc.so.6

  ====================================

backtrace

===================================

#0  0x08e8e206 in raise () from /lib/libc.so.6

#1  0x08e8fbd1 in abort () from /lib/libc.so.6

#2  0x08497bf1 in IntentionalAbort (reason=0x86fcb88 "SDL Router Services declared dead. This may be due to high CPU usage or blocked function. Attempting to restart

CTIManager.") at ProcessCTIProcMon.cpp:65

#3  0x08497d0c in CMProcMon::verifySdlTimerServices () at ProcessCTIProcMon.cpp:573

#4  0x084989b8 in CMProcMon::callManagerMonitorThread (cmProcMon=0xa79c800) at ProcessCTIProcMon.cpp:330

#5  0x0662eca7 in ACE_OS_Thread_Adapter::invoke (this=0xacfb2d0) at OS_Thread_Adapter.cpp:94

#6  0x065e4541 in ace_thread_adapter (args=0xacfb2d0) at Base_Thread_Adapter.cpp:137

#7  0x007796e1 in start_thread () from /lib/libpthread.so.0

#8  0x08f3ad3e in clone () from /lib/libc.so.6

====================================

info registers

====================================

eax            0x00

ecx            0xd783448

edx            0x66

ebx            0x1a196681

esp            0x3a19d340x3a19d34

ebp            0x3a19d3c0x3a19d3c

esi            0x3a19ddc60923356

edi            0x8fbfff4150732788

eip            0x8e8e2060x8e8e206 <raise+70>

eflags         0x200202[ IF ID ]

cs             0x73115

ss             0x7b123

ds             0xc040007b-1069547397

es             0xc040007b-1069547397

fs             0x00

gs             0x3351

====================================

info threads

====================================

  25 Thread 0xb7f2c6d0 (LWP 6681)  0x0077a8e2 in pthread_join () from /lib/libpthread.so.0

  24 Thread 7145  0x08ef98ec in nanosleep () from /lib/libc.so.6

  23 Thread 7186  0x08f30d29 in poll () from /lib/libc.so.6

  22 Thread 7211  0x08f33707 in select () from /lib/libc.so.6

  21 Thread 7342  0x08f33707 in select () from /lib/libc.so.6

  20 Thread 7431  0x0077d9f0 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0

  19 Thread 7432  0x08ef98ec in nanosleep () from /lib/libc.so.6

  18 Thread 12879  0x0077dd08 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0

  17 Thread 12880  0x08f30d29 in poll () from /lib/libc.so.6

  16 Thread 3059  0x08ef98ec in nanosleep () from /lib/libc.so.6

  15 Thread 3060  0x0077d9f0 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0

  14 Thread 3061  0x0077d9f0 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0

  13 Thread 3062  0x08f2b191 in read () from /lib/libc.so.6

  12 Thread 3063  0x08f2b191 in read () from /lib/libc.so.6

  11 Thread 3064  0x0077d9f0 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0

  10 Thread 3066  0x0077d9f0 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0

  9 Thread 3445  0x08ef98ec in nanosleep () from /lib/libc.so.6

  8 Thread 3446  stlp_std::hashtable<stlp_std::pair<unsigned int const, CtiFeatActRegState>, unsigned int, stlp_std::hash<unsigned int>,

stlp_priv::_HashMapTraitsT<stlp_std::pair<unsigned int const, CtiFeatActRegState> >, stlp_priv::_Select1st<stlp_std::pair<unsigned int const, CtiFeatActRegState> >,

stlp_std::equal_to<unsigned int>, stlp_std::allocator<stlp_std::pair<unsigned int const, CtiFeatActRegState> > >::clear (this=0xb440bb08) at /view/BLD-cm_respin_9_1_1

-raw-d/vob/ccm_tpl/release/include/stlport/stl/_hashtable.c:430

  7 Thread 3447  0x08f2b191 in read () from /lib/libc.so.6

  6 Thread 3449  0x08ef98ec in nanosleep () from /lib/libc.so.6

  5 Thread 3450  0x08f2b191 in read () from /lib/libc.so.6

  4 Thread 3505  0x08f2b191 in read () from /lib/libc.so.6

  3 Thread 3509  0x08f2b191 in read () from /lib/libc.so.6

  2 Thread 4015  0x08f30d29 in poll () from /lib/libc.so.6

* 1 Thread 0x3a1ab90 (LWP 3448)  0x08e8e206 in raise () from /lib/libc.so.6

7 REPLIES
New Member

CUCM 9.1.1a CTIManager service core dump

Hi,

Looks like CTI Manager cored due to high CPU usage on system.

To go further I would suggest to take for the core dump occurence time stamp using RTMT:

1)CTI Managager SDI/SDL traces

2)Perfmon Logs

3)Event Viewer Application Logs / System Logs

If possible, go ahead and open a TAC SR for further investigation too.

Regards,
Avinash

Regards, Avinash
New Member

Hi All,

Hi All,

The same issue happened with CUCM 9.1.2. I opened a case with TAC last week. We checked everything related to UCS hardware and compatibility, they are all good. TAC engineer suggested we need to go for RMA regarding the Bug ID CSCua79544.

The odd thing is that all core dump issues are happened after 8pm. I'm planning to increase memories and HDD storage to see what happens in the next couple weeks.

Thanks,

Eric

New Member

Hi Eric,

Hi Eric,

Not sure if you resolved this issue or not - we've got exactly same issue only we are on  9.1.2.11900-12. Another similar bug (CSCud76724) should've been fixed in this version according to bug toolkit but clearly not.

 

Did you find a useful resolution?

 

Rich

New Member

Rich,

Rich,

The issue have not solved yet. The TAC engineer suggested us to go for RMA to replace the existing BE6KM. Since all UC apps need to be installed on the new BE6K, so we have not decided yet.

The CUCM version is 9.1.2.10000-28.

Eric

New Member

Hi Erick, Have you found any

Hi Erick,

 

Have you found any resolution, we found the same issue in our environment with the CUCM version 9.1. Thanks in advance.

 

Regards,

Das

Cisco Employee

Hi Das,It would be best to

Hi Das,

It would be best to open a TAC case with the logs requested by Avinash below, additionally if you are running cucm 9.1.1 then an upgrade to latest 9.1.2 version should be considered.

New Member

Hi Manish,We already have a

Hi Manish,

We already have a case opened and shared the required logs. They suspect this issue is because of the UCS problem. Still waiting for the update , have you heard anything regarding the Bug ID CSCua79544. I already gone through the below links..

https://tools.cisco.com/bugsearch/bug/CSCua79544/?referring_site=bugquickviewredir

http://www.cisco.com/c/en/us/support/docs/unified-communications/unified-communications-manager-callmanager/118702-technote-cucm-00.html

 

Kindly let us know if anyone know the exact workaround on UCS servers.

 

Regards,

Das

694
Views
0
Helpful
7
Replies
CreatePlease login to create content