cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
13551
Views
35
Helpful
0
Comments
Andrew Gossett
Cisco Employee
Cisco Employee

 

Introduction

 

With the increasing complexity of networking devices and protocols, it can be extremely difficult to pinpoint the source of a networking problem.  Often we need to determine if a frame was received and forwarded correctly on a particular device.  There are several capturing tools, debugs, and tricks available to help answer this question.  However, not all are feasible or available to run on a production network.

 

ELAM (Embedded Logic Analyzer Module) is an engineering tool that gives us the ability to look inside Cisco ASICs and understand how a packet is being forwarded.  ELAM is “embedded” within the forwarding pipeline and can capture a packet in real time without affecting performance or control plane resources.  It can help answer questions like:

 

  1. Did the packet reach the forwarding engine?
  2. On what port and VLAN was the packet received?
  3. What did the packet look like (L2 – L4 data)?
  4. How was the packet altered and where was it sent?

 

And much more…

 

ELAM is extremely powerful, granular, and non-intrusive.  It is an extremely valuable troubleshooting tool for Cisco TAC engineers who work on hardware switching platforms.

 

ELAM Challenges

 

ELAM was designed as a diagnostic tool for internal use.  The CLI syntax utilizes internal code names for Cisco ASICs and interpreting the ELAM data requires some hardware specific architecture and forwarding knowledge.  Many of these details cannot be explained because they expose the internal Cisco proprietary features that make Cisco devices best in class.

 

For these reasons, ELAM is not a customer-supported feature and has remained a diagnostic tool for internal use.  There are no external configuration guides and the syntax and operation may change from version to version without any notice.

 

So, with such a firm disclaimer and challenges, why are we discussing ELAM now? 

 

First, it is very common that a TAC engineer will utilize ELAM to isolate an issue. TAC may even request you to perform ELAM if the issue is intermittent.  It’s important to understand that these steps are non-intrusive and how they can help provide a root cause analysis. 

 

Also, there are times when there are no other tools are available that can help isolate an issue (i.e., no configuration changes allowed during production hours for SPAN, ACL hits, intrusive debugs, etc…).  There may not be time to get TAC on the line and ELAM can be an extremely helpful tool to have in your back pocket as a last resort.

 

ELAM Basics

 

ELAM can be performed without full architecture knowledge of each platform.  This section will discuss the basics needed to perform an ELAM on the Catalyst 6500 and 7600 platforms along with the Nexus 7000.

 

 

ELAM Workflow

 

As previously discussed, ELAM is dependent on the underlining hardware and therefore the CLI syntax will be dependent on the hardware in use.  However, each platform will follow a similar workflow as shown below.  Please refer to the ELAM Examples section to see how this workflow is applied on different platforms.

elam_workflow.png

 

 

      1. Identify the expected ingress-forwarding engine (FE).  When platforms have more than one FE, it is critical to identify which FE makes the forwarding decision for the packet you are trying to capture.  Configure the ELAM on the correct FE.
      2. Configure the ELAM trigger.  You will need to configure a trigger with details specific to the packet you are trying to capture.  Common triggers include a source and destination IP address or L4 port numbers.  ELAM allows for multiple fields to be specified and will perform a logical AND on all fields configured.
      3. Start the ELAM
      4. Wait for the ELAM to trigger and display the result.

 

 

Centralized vs. Distributed Forwarding

 

The first step in performing an ELAM is to identify the correct forwarding engine (FE).  A 6500 with classical or centralized forwarding (CFC) linecards utilizes centralized forwarding where the active supervisor makes the forwarding decision.  For packets that ingress on classical or CFC linecards, you will perform the ELAM on the active supervisor.

 

With distributed forwarding (DFC’s) enabled linecards, the forwarding decision is made locally by an FE on the linecard without sending to the supervisor.  For packets that ingress DFC linecards, you will perform the ELAM on the linecards itself.

 

For the Nexus 7000, all linecards are fully distributed.  Additionally, most linecards have multiple FEs.  When setting up the ELAM, you will need to know which port the packet is received on and determine the FE that maps to that port.

 

Please refer here for more information on hardware and forwarding architecture:

 

BRKARC-3465 Cisco Catalyst 6500 Switch Architecture

https://ciscolive365.com/connect/sessionDetail.ww?SESSION_ID=7674

 

BRKARC-3470 - Cisco Nexus 7000 Switch Architecture

https://ciscolive365.com/connect/sessionDetail.ww?SESSION_ID=7677

 

 

Data Bus (DBUS) and Result Bus (RBUS)

The DBUS contains information used by the FE to make a forwarding decision.  It contains several platform specific internal fields along with the header information for a frame.  Viewing the DBUS helps determine where the packet was received and it's L2-L4 information.

 

The RBUS contains the forwarding decision made by the FE.  Viewing the RBUS helps determine if the frame was altered and where it was sent.

 

Local Target Logic (LTL)

The LTL is an index used to represent a port or group of ports.  The source LTL index and the destination LTL index tell us where the frame was received and where it was sent.  Different platforms and supervisors will have different commands to decode the LTL values.

 

Flood Bit

LTL values are displayed as 5 or less hex numbers (i.e., 0xa2c).  The flood bit is the 16th bit in the LTL result.  Often the RBUS will display a field with the destination LTL index and a separate field for the flood bit.  It is important to merge these results for the correct LTL.  For example:

 

RBUS data:

FLOOD ........................... [1] = 1

DEST_INDEX ...................... [19] = 0x48

 

In this example, the destination LTL index is 0x48.  Since the flood bit is one, we need to set the 16th bit in the LTL to 1:

 

0x00048 = 0000 0000 0000 0100 1000

               |

               +---- Flood bit, set to 1 = 0x08048

 

 

After accounting for the flood bit, the destination index has become 0x8048

 

ELAM Examples

 

The purpose of these examples is to show how ELAM can be used to validate basic IPv4 or IPV6 unicast flows.  As described in the ELAM Challenges section, it is not practical to explain all internal fields or packet types (i.e., recirculation for multicast, tunnels, MPLS, etc…). 

 

 

  1. ELAM Example: Sup720
  2. ELAM Example: Sup2T
  3. ELAM Example: Nexus7000 M-series
  4. ELAM Example: Nexus7000 F1
  5. ELAM Example: Nexus7000 F2

 

As a reference, the internal ASIC name used for the ELAM is listed below for each module type.

 

PlatformModule TypeInternal ASIC Name

Catalyst 6500/ Cisco 7600Sup720 (PFC3, DFC3)Superman

Catalyst 6500

Sup2T (PFC4, DFC4)Eureka
Nexus 7000M-SeriesEureka
Nexus 7000F1Orion
Nexus 7000F2Clipper

 

 

Is there a more customer friendly way to utilize ELAM?

 

Yes.  Starting in 12.2(50)SY, IOS added the “show platform datapath” command for the Catalyst 6500 Sup2T.  This command utilized ELAM to capture and display the forwarding result of a specific packet.

 

For Nexus7000, an easy to use script “elame” was added in 6.2(2) to leverage ELAM:

N7KA# source sys/elame

elam helper, version 1.015

Usage:

elame [<src>] <dest> [vlan <vlan#>] [vrf <vrf_name>] [int <interface> | vdc] [trace]

  where

   <src>, <dest> are ipv4 addresses in form 1.2.3.4

   <vlan>, <interface> - indicate ingress vlan/interface

   vdc - use all elams in current vdc

   trace - keep record of all outputs in volatile:elame.log

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: