Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for
Search instead for
Did you mean:
Annotating Troubleshooting Sessions
When capturing log sessions during an outage or debug session it's crucial to keep a history of what was done when and in what order. This helps in post mortem analysis after the event or session is over. There are a couple of standard things to do to help this such as:
NTP time sync all devices in the network to get them on a single time sourceEnable syslog for all devices in the network Allocate some local log space on each local device based on a percentage of free memory available on the deviceAlways have trend graphs from some polling mechanism for CPU levels, memory usage, traffic rates, etc.
In addition to these here are a few other tips that really help TAC analyze log and debug sessions that are collected:
Annotate the local log and syslog output using thesend logcommand
CE_#send log "Shutting down interface now!"CE_#config tEnter configuration commands, one per line.End with CNTL/Z.CE_(config)#int e 0/0CE_(config-if)#shutCE_(config-if)#endCE_#sh log%SYS-7-USERLOG: Message from 0(): "Shutting down interface now!"00:00:46: %SYS-5-CONFIG_I: Configured from console by console00:00:47: %LINK-5-CHANGED: Interface Ethernet0/0, changed state toadministratively down00:00:48: %LINEPROTO-5-UPDOWN: Line protocol on Interface Ethernet0/0,changed state to down
By using thesend log <message>the output is sent to the local log as well as to the syslog so there is a historical view of what was done and why. View it as "Thinking out loud and capturing the thought process" so it can be followed later. Without it it's very hard to remember exactly what was done when and why.
Annotate the logging or debug session as things are learned or events happen using the!<message>CLI
CE_#!Troubleshooting RIP routers being lostCE_#!checking # of routers learned before cable is pulled
CE_#sh ip ro summIP routing table name is default (0x0)IP routing table maximum-paths is 32Route SourceNetworksSubnetsReplicatesOverheadMemory(bytes)connected020104344static00000rip01052172eigrp 100000internal1528Total1301561044CE_#!Only one RIP route...wait some more time for network to converge
Capture date, time, and CPU automatically when running CLI commands during a session by enablingterminal exec prompt timestamp
CE_#term exec prompt timestampCE_#sh userLoad for five secs: 3%/0%; one minute: 1%; five minutes: 1%Time source is hardware calendar, *14:02:40.623 EST Mon Oct 20 2008LineUserHost(s)IdleLocation*0 con 0idle00:00:00InterfaceUserModeIdlePeer AddressCE_#>>
Notice the Load and Time as the output for each command. It will be printed automatically every time a CLI command is entered. This is very useful because it allows deltas of counters between outputs to have some rate context by comparing the time difference between outputs. Without that when looking at a raw capture log it's impossible to tell how much time elapsed between outputs. When using the above in conjunction with "|" don't forget to include the "Load for five|Time source is" to capture it while filtering other output.
To receive the latest information on Cisco online tools, certifications, support documentation, insights from Cisco experts and peers, and upcoming events, check out theCisco Technical Services Newslettertoday.