Innovative Troubleshooting Tools in Cisco Switches

Yogesh Ramdoss · ‎08-29-2013

At Cisco TAC, when the phones ring for P1/P2 outages, a typical problem we hear from customers is about connectivity issues between end-hosts and/or end-users' application performance issues.

In an enterprise core/distribution and datacenter network, where virtual switching solutions (like, VSS or vPC) are highly deployed with large number of port-channels / redundant links, it is challenging and critical to trace the path of the packet to identify the device, link and port causing the above-mentioned problems.

How does a TAC engineer really find the device / link / port in issue ? The answer is ... by using the innovative tools discussed here.

(1) NetDR / EthAnalyzer

Net Driver (NetDR) tool is used in Catalyst 6500 and Cisco 7600 platforms to capture the traffic sent to (and received from) the CPU. EthAnalyzer performs the same functions in Nexus platforms and is an implementation of TShark in NX-OS.

When poor performance is experienced with specific application, using these tools (in the devices along the traffic path) helps us to confirm that specific traffic is getting hardware-switched and NOT processed by the CPU.

Here is a sample NetDR capture - reporting layer2, layer3 and layer4 information along with ingress interface.

(2) Access-List / VLAN Access-List Capture (ACL / VACL Capture)

This tool allows network administrators to replicate network traffic, especially when there is a need to monitor subset of traffic on high bandwidth links (thats how it differs from traditional R/ER/SPAN). VACL Capture is supported only on Catalyst 6500 / Cisco 7600 platforms. ACL Capture is supported only on Nexus7000 M1-series modules in 5.2(1) release onwards.

The examples discussed are very simple but powerful in terms of narrowing down the device(s)

It can be very invaluable if number of packets for a specific flow can be counted at each layer-2 hop, specifically for faster problem resolution.

(3) Embedded Logic Analyzer Module (ELAM)

First of all, ELAM is NOT a supported feature and is very specific to the hardware architecture. Anything and everything about it may change from version to version without any notice. ELAM is supported in Catalyst 6500, Cisco 7600, Nexus 7000 platforms.

ELAM is used by TAC as a capturing tool to validate:

(a) Was the packet received ?

(b) What did the packet look like ?

(c) Was the packet rewritten/alterted ? If so, how ?

(d) Where was the packet sent ?

The following diagram shows the workflow of using the ELAM tool:

As you can see, the first step is to identify the ingress forwarding engine, which requires knowledge on hardware architecure. It is highly recommened for customers to review a detailed document on ELAM available here.

Miscellaneous:

A relevant TS Newsletter Article has discussed different packet capturing tools available in Cisco devices:

http://www.cisco.com/web/services/news/ts_newsletter/archives/201206.html

A Cisco Live 2013 Breakout session BRKARC-2011 discusses and provides an Overview of Troubleshooting Tools in Cisco Switches and Routers.

Session Slides (in PDF)

Session Video (in MP4)

To access contents, please visit http://www.ciscolive365.com and register.

References:

VACL Capture for Granular Traffic Analysis with Catalyst 6500 running Cisco IOS Software:

http://www.cisco.com/en/US/tech/tk389/tk814/technologies_configuration_example09186a00808122ac.shtml

Nexus7000 NX-OS Security Configuration Guide - ACL Capture:

http://www.cisco.com/en/US/docs/switches/datacenter/sw/5_x/nx-os/security/configuration/guide/b_Cisco_Nexus_7000_NX-OS_Security_Configuration_Guide__Release_5.x_chapter_01110.html#con_1210975