- Bronze, 100 points or more
With the increasing complexity of networking devices and protocols, it can be extremely difficult to pinpoint the source of a networking problem. Often we need to determine if a frame was received and forwarded correctly on a particular device. There are several capturing tools, debugs, and tricks available to help answer this question. However, not all are feasible or available to run on a production network.
ELAM (Embedded Logic Analyzer Module) is an engineering tool that gives us the ability to look inside Cisco ASICs and understand how a packet is being forwarded. ELAM is “embedded” within the forwarding pipeline and can capture a packet in real time without affecting performance or control plane resources. It can help answer questions like:
- Did the packet reach the forwarding engine?
- On what port and VLAN was the packet received?
- What did the packet look like (L2 – L4 data)?
- How was the packet altered and where was it sent?
And much more…
ELAM is extremely powerful, granular, and non-intrusive. It is an extremely valuable troubleshooting tool for Cisco TAC engineers who work on hardware switching platforms.
ELAM was designed as a diagnostic tool for internal use. The CLI syntax utilizes internal code names for Cisco ASICs and interpreting the ELAM data requires some hardware specific architecture and forwarding knowledge. Many of these details cannot be explained because they expose the internal Cisco proprietary features that make Cisco devices best in class.
For these reasons, ELAM is not a customer-supported feature and has remained a diagnostic tool for internal use. There are no external configuration guides and the syntax and operation may change from version to version without any notice.
So, with such a firm disclaimer and challenges, why are we discussing ELAM now?
First, it is very common that a TAC engineer will utilize ELAM to isolate an issue. TAC may even request you to perform ELAM if the issue is intermittent. It’s important to understand that these steps are non-intrusive and how they can help provide a root cause analysis.
Also, there are times when there are no other tools are available that can help isolate an issue (i.e., no configuration changes allowed during production hours for SPAN, ACL hits, intrusive debugs, etc…). There may not be time to get TAC on the line and ELAM can be an extremely helpful tool to have in your back pocket as a last resort.
ELAM can be performed without full architecture knowledge of each platform. This section will discuss the basics needed to perform an ELAM on the Catalyst 6500 and 7600 platforms along with the Nexus 7000.
As previously discussed, ELAM is dependent on the underlining hardware and therefore the CLI syntax will be dependent on the hardware in use. However, each platform will follow a similar workflow as shown below. Please refer to the ELAM Examples section to see how this workflow is applied on different platforms.
- Identify the expected ingress-forwarding engine (FE). When platforms have more than one FE, it is critical to identify which FE makes the forwarding decision for the packet you are trying to capture. Configure the ELAM on the correct FE.
- Configure the ELAM trigger. You will need to configure a trigger with details specific to the packet you are trying to capture. Common triggers include a source and destination IP address or L4 port numbers. ELAM allows for multiple fields to be specified and will perform a logical AND on all fields configured.
- Start the ELAM
- Wait for the ELAM to trigger and display the result.
Centralized vs. Distributed Forwarding
The first step in performing an ELAM is to identify the correct forwarding engine (FE). A 6500 with classical or centralized forwarding (CFC) linecards utilizes centralized forwarding where the active supervisor makes the forwarding decision. For packets that ingress on classical or CFC linecards, you will perform the ELAM on the active supervisor.
With distributed forwarding (DFC’s) enabled linecards, the forwarding decision is made locally by an FE on the linecard without sending to the supervisor. For packets that ingress DFC linecards, you will perform the ELAM on the linecards itself.
For the Nexus 7000, all linecards are fully distributed. Additionally, most linecards have multiple FEs. When setting up the ELAM, you will need to know which port the packet is received on and determine the FE that maps to that port.
Please refer here for more information on hardware and forwarding architecture:
BRKARC-3465 Cisco Catalyst 6500 Switch Architecture
BRKARC-3470 - Cisco Nexus 7000 Switch Architecture
Data Bus (DBUS) and Result Bus (RBUS)
The DBUS contains information used by the FE to make a forwarding decision. It contains several platform specific internal fields along with the header information for a frame. Viewing the DBUS helps determine where the packet was received and it's L2-L4 information.
The RBUS contains the forwarding decision made by the FE. Viewing the RBUS helps determine if the frame was altered and where it was sent.
Local Target Logic (LTL)
The LTL is an index used to represent a port or group of ports. The source LTL index and the destination LTL index tell us where the frame was received and where it was sent. Different platforms and supervisors will have different commands to decode the LTL values.
LTL values are displayed as 5 or less hex numbers (i.e., 0xa2c). The flood bit is the 16th bit in the LTL result. Often the RBUS will display a field with the destination LTL index and a separate field for the flood bit. It is important to merge these results for the correct LTL. For example:
FLOOD ...........................  = 1
DEST_INDEX ......................  = 0x48
In this example, the destination LTL index is 0x48. Since the flood bit is one, we need to set the 16th bit in the LTL to 1:
0x00048 = 0000 0000 0000 0100 1000
+---- Flood bit, set to 1 = 0x08048
After accounting for the flood bit, the destination index has become 0x8048.
The purpose of these examples is to show how ELAM can be used to validate basic IPv4 or IPV6 unicast flows. As described in the ELAM Challenges section, it is not practical to explain all internal fields or packet types (i.e., recirculation for multicast, tunnels, MPLS, etc…).
- ELAM Example: Sup720
- ELAM Example: Sup2T
- ELAM Example: Nexus7000 M-series
- ELAM Example: Nexus7000 F1
- ELAM Example: Nexus7000 F2
As a reference, the internal ASIC name used for the ELAM is listed below for each module type.
|Platform||Module Type||Internal ASIC Name|
|Catalyst 6500/ Cisco 7600||Sup720 (PFC3, DFC3)||Superman|
|Sup2T (PFC4, DFC4)||Eureka|
Is there a more customer friendly way to utilize ELAM?
Yes. Starting in 12.2(50)SY, IOS added the “show platform datapath” command for the Catalyst 6500 Sup2T. This command utilized ELAM to capture and display the forwarding result of a specific packet.
For Nexus7000, an easy to use script “elame” was added in 6.2(2) to leverage ELAM:
N7KA# source sys/elame
elam helper, version 1.015
elame [<src>] <dest> [vlan <vlan#>] [vrf <vrf_name>] [int <interface> | vdc] [trace]
<src>, <dest> are ipv4 addresses in form 22.214.171.124
<vlan>, <interface> - indicate ingress vlan/interface
vdc - use all elams in current vdc
trace - keep record of all outputs in volatile:elame.log