Several Cisco TAC engineers collaborated on the answer to this question sent in by TS Newsletter readers Larry Dutton and Yasser Slarmie.
Tools usage to solve networking issues is a common question among customers. In TAC we do have access to some proprietary tools but most of them are for working with source code and working on bug related issues. We try very hard to expose all possible tools directly to the customer to enable them to troubleshoot problems directly.
Some examples are:
Enhanced Packet Capture, which was originated inside TAC and pushed external based on its success:http://www.cisco.com/go/epc
We also use the Embedded Event Manager, which is included in almost all IOS devices by default, to automate collection of troubleshooting data at the moment a problem occurs. Simple EEM applets can be used to simplify data collection and identify problems much more quickly. For example, if a low memory condition is detected, this applet can be used to collect all of the necessary troubleshooting data TAC needs to diagnose the problem.
event manager applet LOW_IO_MEM
event snmp oid 220.127.116.11.18.104.22.168.22.214.171.124.6.2 get-type exact entry-op lt entry-val 1000000 entry-type value exit-op gt exit-val 100000000 exit-type value poll-interval 10
action 0.0 syslog msg "LOW I/O MEMORY DETECTED. Please wait - logging information to flash:low_mem.txt”
action 0.1 cli command "enable"
action 0.2 cli command "term exec prompt timestamp”
action 1.2 cli command "show memory statistics | append flash:low_mem.txt"
action 1.3 cli command "show process cpu sorted | append flash:low_mem.txt"
action 1.5 cli command "show interfaces | append flash:low_mem.txt"
action 1.6 cli command "show interfaces stat | append flash:low_mem.txt"
action 1.7 cli command "show ip traffic | append flash:low_mem.txt”
action 2.2 cli command "show buffers | append flash:low_mem.txt"
action 2.3 cli command "show buffers failures | append flash:low_mem.txt"
action 2.4 cli command "show buffers assigned dump | append flash:low_mem.txt”
action 3.2 cli command "show log | append flash:low_mem.txt"
action 3.3 cli command "show tech | append flash:low_mem.txt"
action 3.4 cli command "show start | append flash:low_mem.txt”
action 4.2 cli command "show interfaces | append flash:low_mem.txt"
action 4.3 cli command "show interfaces stat | append flash:low_mem.txt"
action 4.4 cli command "show ip traffic | append flash:low_mem.txt"
Checkout Cisco Beyond at for more EEM examples contributed by Cisco employees and our customers and partners.
We use many open source and publicly-available tools to diagnose and solve customer network problems as well! Here are just some of the tools we use every day:
- netcat - TCP/IP Swiss army knife. Used for simulating clients and servers
- tcpreplay/tcprewrite - packet replay and modification software. Great for replaying packet captures supplied by customers to try and reproduce problems
- SecureCRT, PUTTY - SSH clients used to securely connect to Cisco device Command Line Interfaces (CLI)
- Wireshark - Packet capture analyzer
- mrtg - SNMP graphing program
- Net-SNMP toolchain for testing SNMP access to devices
- nix command line tools: grep, awk, sed
We also have various test suites in our labs such as Spirent test center and Ixia to do things like packet generation, route injection, etc.
In order to determine what SNMP management capability one can expect from Cisco devices, we consult the SNMP Object Navigator and MIB Locator tools on Cisco.com. Checkout http://www.cisco.com/go/mibs for a one-stop place for SNMP support and MIB download information.
Also check out this ASA podcast for more tool information:https://supportforums.cisco.com/docs/DOC-12641
One of the key benefits we have is a mass of cases where we can search to find similar problems. Similar in concept to the discussions and documents on Cisco.com and the Support Forums.
Lastly, and probably one of the most valuable tools we use, is the people. While not considered directly as tools, the crowd wisdom of engineers helping engineers is by far the most powerful asset in TAC. Being able to bounce ideas off of other engineers in a similar area, learn by participating, etc. Our case-handling tools are always being improved to encourage collaboration between engineers to collectively solve problems.
TS Newsletter Editor's Note: If you have a question for a Cisco TAC engineer, send it to firstname.lastname@example.org. If we publish your question, you’ll be the envy of all your friends, and we’ll send you some free stuff...
Subscribe to the TS Newsletter.