Cisco Support Community
cancel
Showing results for 
Search instead for 
Did you mean: 

Annotating Troubleshooting Sessions

When capturing log sessions during an outage or debug session it's crucial to keep a history of what was done when and in what order. This helps in post mortem analysis after the event or session is over. There are a couple of standard things to do to help this such as:

NTP time sync all devices in the network to get them on a single time source
Enable syslog for all devices in the network 
Allocate some local log space on each local device based on a percentage of free memory available on the device
Always have trend graphs from some polling mechanism for CPU levels, memory usage, traffic rates, etc.

In addition to these here are a few other tips that really help TAC analyze log and debug sessions that are collected:

  • Annotate the local log and syslog output using the send log command
CE_#send log "Shutting down interface now!"
CE_#config t                                     
Enter configuration commands, one per line.  End with CNTL/Z. 
CE_(config)#int e 0/0                                        
CE_(config-if)#shut                                    
CE_(config-if)#end                                   
CE_#sh log                                                      
%SYS-7-USERLOG: Message from 0(): "Shutting down interface now!"    
00:00:46: %SYS-5-CONFIG_I: Configured from console by console      
00:00:47: %LINK-5-CHANGED: Interface Ethernet0/0, changed state to  
administratively down                                                           
00:00:48: %LINEPROTO-5-UPDOWN: Line protocol on Interface Ethernet0/0, 
changed state to down

By using the send log <message> the output is sent to the local log as well as to the syslog so there is a historical view of what was done and why. View it as "Thinking out loud and capturing the thought process" so it can be followed later. Without it it's very hard to remember exactly what was done when and why.

  • Annotate the logging or debug session as things are learned or events happen using the !<message> CLI
CE_#!Troubleshooting RIP routers being lost                     
CE_#!checking # of routers learned before cable is pulled       
CE_#sh ip ro summ                                               
IP routing table name is default (0x0)                            
IP routing table maximum-paths is 32                                 
Route Source    Networks    Subnets     Replicates  Overhead    Memory   
(bytes)                                                                         
connected       0           2           0           104         344             
static          0           0           0           0           0               
rip             0           1           0           52          172             
eigrp 1         0           0           0           0           0               
internal        1                                               528             
Total           1           3           0           156         1044            
CE_#!Only one RIP route...wait some more time for network to converge

  • Capture date, time, and CPU automatically when running CLI commands during a session by enabling terminal exec prompt timestamp
CE_#term exec prompt timestamp                                                  
CE_#sh user                                                                     
Load for five secs: 3%/0%; one minute: 1%; five minutes: 1%                     
Time source is hardware calendar, *14:02:40.623 EST Mon Oct 20 2008             
                                                                                
    Line       User       Host(s)              Idle       Location              
*  0 con 0                idle                 00:00:00                         
                                                                               
  Interface    User               Mode         Idle     Peer Address            
                                                                               
CE_#  >> 

Notice the Load and Time as the output for each command. It will be printed automatically every time a CLI command is entered. This is very useful because it allows deltas of counters between outputs to have some rate context by comparing the time difference between outputs. Without that when looking at a raw capture log it's impossible to tell how much time elapsed between outputs. When using the above in conjunction with "|" don't forget to include the "Load for five|Time source is" to capture it while filtering other output.

To receive the latest information on Cisco online tools, certifications, support documentation, insights from Cisco experts and peers, and upcoming events, check out the Cisco Technical Services Newsletter today.

Version history
Revision #:
1 of 1
Last update:
‎07-03-2009 12:42 AM
Updated by:
 
Labels (1)
Everyone's tags (2)
Comments
New Member

thanks for sharing. good job. F85 Treadmill