cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
41910
Views
75
Helpful
10
Comments
Sreekanth Narayanan
Cisco Employee
Cisco Employee

We have a Multi-Services Voice Debug Lookup Tool for different troubleshooting scenarios located here: www.cisco.com/en/US/tech/tk1077/technologies_tech_note09186a0080207ec6.shtml


This document indicates the different combination of traces  that are required by TAC in different scenarios so that customers can  provide complete information to the CSE at the beginning of the case  itself. This saves time and allows us to solve the cases faster as  majority (if not all) of the right information is given to us, and we do  not need to ask for it again. This document only mentions the traces  required from the call manager side of the issue. If other applications  such as Attendant Console, UCCX, gateways are involved, we will need the  respective traces from there as well.


There is a Cisco Support Forums document that indicates the  locations of the traces for different services in call manager. This can  be used in case the RTMT tool is not functioning as expected, and when  solving the main problem is a priority. This document is located here: supportforums.cisco.com/docs/DOC-16943


NOTE 1: Please make sure that the traces for the  services in question (e.g. Cisco Call Manager, Cisco Extension Mobility,  Cisco Dirsync) are at the highest level possible during the occurence  of issue or when re-creating the issue. Traces not set to the highest  level are incomplete.


I have specified a few scenarios below that are commonly faced by the customer. This can be used as a reference.

NOTE 2: The required traces have to be collected from all  servers in the cluster to account for SDL communication between the  servers for call processing or media resources.

I am attaching a few images here of the traces that are usually collected from the "Collect Files" option in RTMT.

PAGE I

imptraces.jpg

imprtraces2.jpg

PAGE II

imptracespage2.jpg

imptracespage2-1.jpg

imptracespage2-2.jpg

imptracespage2-3.jpg

    1.CALL ROUTING

         a. Detailed Cisco Call Manager Traces

         Please make sure that the SIP Call Processing Traces and  SIP Stack traces are also checked in case you have any SIP  devices/trunks involved in the call flow. SIP Stack traces should be  enabled when the call volume is not very high as it takes up more CPU  cycles, leaving less available for call processing.

         SoftKey traces are also important and can be turned on.

         We don't need to change the level of SDL traces, unless explicitly asked for.

         Scrshotj.jpg

         b. Event Viewer - Application and Event Viewer - System logs.

         c. Timestamps of call failures, Calling and Called numbers.

         d. Detailed Call Flow (Devices involved in the call flow & Protocols).

    2.HIGH CPU/MEMORY UTILIZATION

         Please collect all of the below traces for a particular  time period before the problem began till after the problem went away.  For instance, if we started observing high CPU or memory usage at 4 P.M.  and the problem went away by 5 P.M., then we would collect traces for a  time interval of 3:30 P.M. to 5:30 P.M. This time interval can vary  from one issue to another, and also based on the customer set up.

         a. Detailed Cisco Call Manager traces.

         b. Event Viewer - Application and Event Viewer - System logs.

         c. Cisco RISDC and Perfmon logs.

         d. Cisco AMC service.

         e. Cisco Tomcat and Tomcat Security logs.

         f. Outputs from CLI: 'show status', 'show process using-most cpu/memory', 'show process load'.

         g. Output of 'utils diagnose test'.

    3. EXTENSION MOBILITY LOGINS

         a. Detailed Cisco Call Manager traces.

         b. Cisco Extension Mobility.

         c. Cisco Extension Mobility Application.

         d. Cisco Tomcat and Tomcat Security logs.

    4. AGENT LOGINS - UCCX

        a. Detailed Cisco Call Manager traces.

        b. Cisco Extension Mobility.

        c. Cisco Extension Mobility Application.

        d. Cisco Tomcat and Tomcat Security logs.

        e. Cisco AXL Web Service.

        f. Detailed Cisco CTI Manager logs.

    5.LDAP SYNCHRONIZATION/AUTHENTICATION

        a. Cisco Dirsync logs.

        b. Cisco Tomcat and Tomcat Security logs.

        c. Packet Captures on the call manager server when the sync is initialized also help.

    6.DRF BACKUP

       a. Cisco DRF Master.

       b. Cisco DRF Local.

       c. Failure logs from the DRF 'Current Status' page.

    7. PHONE REGISTRATION

        a. Detailed Cisco Call Manager traces.

        b. Packet captures from both sides - CUCM server and IP Phone or the switch port to which the IP Phone is connected.

        c. Event Viewer - Application and Event Viewer - System logs.

        d. Phone Status Messages.

        e. Phone Console Logs.

    8.IPMA

        a. Detailed Cisco Call Manager traces.

        b. Detailed Cisco CTI Manager traces.

        c. Cisco IP Manager Assistant.

        d. JTAPI logs from the client side - PC.

    9. CAR/CDR

        a. Cisco CAR Scheduler.

        b. Cisco CAR Web Agent.

        c. Cisco CDR Agent.

        d. Cisco CDR Repository Manager

   10. SECURITY - Such as IP Phone rejected due to security errors/ TLS Connections not setting up.

        a. Detailed Cisco Call Manager traces.

        b. Cisco CTL Provider.

        c. Cisco Trust Verification Service.

        d. Cisco Certificate Authority Proxy Function.

   11. MEDIA RESOURCES - MOH not working/ DTMF issues/ No Ringback/ Conference drops/ MTP or XCoder Allocation failures

        a. Detailed Cisco Call Manager traces.

        b. Cisco IP Voice Media Streaming App.

        c. Event Viewer - Application and Event Viewer - System logs.

   12.COREDUMP

        a. utils core active list.

        b. utils core active analyze <coredump name>.

        c. Detailed Cisco Call Manager traces for a particular period of time before the core was created.

        d. Event Viewer - Application and Event Viewer - System logs.

        e. Cisco RISDC and Perfmon Logs for a particular period of time before the core was created.

.       f. Outputs of 'show status', 'utils diagnose test', 'show process load', 'show process using-most memory/cpu'

   13. GUI DISPLAYS WRONG STATUS INFORMATION

        a. Cisco CCMAdmin Web Service.

        b. Cisco Common User Interface.

        c. Cisco Database Layer Monitor.

        d. Cisco Database Notification Service.

        e. Cisco RisBean Library.

        f. Cisco Tomcat.

   14. TOMCAT ISSUES

         a. Cisco Tomcat and Tomcat Security logs.

         b. Output of 'utils diagnose test'. This may generate a heap dump, which you can then collect using 'file get activelog tomcat/logs/*'

         c. Detailed Cisco Call Manager traces.

Comments

Nice one.

Somnath Gupta
Cisco Employee
Cisco Employee

Good One!

Sreekanth Narayanan
Cisco Employee
Cisco Employee

Thanks guys!

Amit Divekar
Cisco Employee
Cisco Employee

Very well done Sreekanth, would definately save a lot of time

Gagarin Sathiyanarayanan
Cisco Employee
Cisco Employee

Hey Sree,

Can we also include the logs to be collected for the 'Corporate Directory' issues which is quite common.

Logs required :

+ CCMCIP logs (Debug)

+ Simultaneous Pcaps from the IP phone and CUCM

+ Cisco Tomcat Logs

 

Regards,

Gagarin

adbalasu
Cisco Employee
Cisco Employee

Very useful reference material !

good Job !

dwasserzug1
Level 1
Level 1

This is a very good article. Here is another set of logs that I find useful. Hopefully this will help others.

Jabber Login Issues

a. Cisco Client Profile Agent

b. Cisco XCP Connection Manager

c. Cisco XCP Authentication Service

d. Cisco XCP Router

e. Cisco Tomcat Security Logs

nurhat19921
Level 1
Level 1

HI! Can you write the list of books about CUCM Trace ? Please! Thanks!

dwasserzug1
Level 1
Level 1

Hello.

Here are some resources that I find helpful:

http://my.safaribooksonline.com/book/telephony/1587050757/understanding-the-troubleshooting-tools/ch03lev1sec3

http://www.cisco.com/c/en/us/td/docs/voice_ip_comm/connection/10x/serv_administration/guide/10xcucservagx/10xcucservag030.pdf

http://www.cisco.com/c/en/us/td/docs/voice_ip_comm/cucm/im_presence/configAdminGuide/10_5_2/CUP0_BK_CEB3E82E_00_config-admin-guide-imp-1052/CUP0_BK_CEB3E82E_00_config-admin-guide-imp-1052_chapter_010011.pdf

Thank you,

Dominic

pkinane
Cisco Employee
Cisco Employee
I made a video about how to setup jEdit for reading logs so that others can see it is a great tool after you optimize it for reading logs. https://www.youtube.com/watch?v=AXslHfpJ5HM Let me know what you think.
Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: