When capturing log sessions during an outage or debug session it's crucial to keep a history of what was done when and in what order. This helps in post mortem analysis after the event or session is over. There are a couple of standard things to do to help this such as:
NTP time sync all devices in the network to get them on a single time source
Enable syslog for all devices in the network
Allocate some local log space on each local device based on a percentage of free memory available on the device
Always have trend graphs from some polling mechanism for CPU levels, memory usage, traffic rates, etc.
In addition to these here are a few other tips that really help TAC analyze log and debug sessions that are collected:
- Annotate the local log and syslog output using the send log command
CE_#send log "Shutting down interface now!"
CE_#config t
Enter configuration commands, one per line. End with CNTL/Z.
CE_(config)#int e 0/0
CE_(config-if)#shut
CE_(config-if)#end
CE_#sh log
%SYS-7-USERLOG: Message from 0(): "Shutting down interface now!"
00:00:46: %SYS-5-CONFIG_I: Configured from console by console
00:00:47: %LINK-5-CHANGED: Interface Ethernet0/0, changed state to
administratively down
00:00:48: %LINEPROTO-5-UPDOWN: Line protocol on Interface Ethernet0/0,
changed state to down
By using the send log <message> the output is sent to the local log as well as to the syslog so there is a historical view of what was done and why. View it as "Thinking out loud and capturing the thought process" so it can be followed later. Without it it's very hard to remember exactly what was done when and why.
- Annotate the logging or debug session as things are learned or events happen using the !<message> CLI
CE_#!Troubleshooting RIP routers being lost
CE_#!checking # of routers learned before cable is pulled
CE_#sh ip ro summ
IP routing table name is default (0x0)
IP routing table maximum-paths is 32
Route Source Networks Subnets Replicates Overhead Memory
(bytes)
connected 0 2 0 104 344
static 0 0 0 0 0
rip 0 1 0 52 172
eigrp 1 0 0 0 0 0
internal 1 528
Total 1 3 0 156 1044
CE_#!Only one RIP route...wait some more time for network to converge
- Capture date, time, and CPU automatically when running CLI commands during a session by enabling terminal exec prompt timestamp
CE_#term exec prompt timestamp
CE_#sh user
Load for five secs: 3%/0%; one minute: 1%; five minutes: 1%
Time source is hardware calendar, *14:02:40.623 EST Mon Oct 20 2008
Line User Host(s) Idle Location
* 0 con 0 idle 00:00:00
Interface User Mode Idle Peer Address
CE_# >>
Notice the Load and Time as the output for each command. It will be printed automatically every time a CLI command is entered. This is very useful because it allows deltas of counters between outputs to have some rate context by comparing the time difference between outputs. Without that when looking at a raw capture log it's impossible to tell how much time elapsed between outputs. When using the above in conjunction with "|" don't forget to include the "Load for five|Time source is" to capture it while filtering other output.
To receive the latest information on Cisco online tools, certifications, support documentation, insights from Cisco experts and peers, and upcoming events, check out the Cisco Technical Services Newsletter today.
