This document provides the steps to perform an ELAM on the Nexus 7000 F2 modules, explains the most relevant outputs, and how to interpret their results. Please refer to the following document for an overview on ELAM:
Topology
In this example a host on Vlan10 (10.1.1.101) port Eth6/4 sends an ICMP request to a host also on Vlan10 (10.1.1.102) off port Eth6/3. We will use ELAM to capture this single packet between the hosts. It's important to remember that ELAM allows us to capture a single frame.
To perform an ELAM on the Nexus7000, you need to first attach to the appropriate module. This requires the network-admin privilege.
N7K# attach module 6
Attaching to module 6 ...
To exit type 'exit', to abort type '$.'
module-6#
Determine the ingress FE
We expect the traffic to ingress the switch on port Eth6/4. Checking the modules in the system we can see that module 6 is an F2 module. Remember, the Nexus 7000 is fully distributed and the modules, not the supervisor, are responsible for making the forwarding decision for dataplane traffic.
N7K# show module 6
Mod Ports Module-Type Model Status
--- ----- ----------------------------------- ------------------ ----------
6 48 1/10 Gbps Ethernet Module N7K-F248XP-25E ok
For F2 modules, we want to perform the ELAM on the L2 forwarding engine (FE) with internal codename Clipper. Note that the L2 FE data bus (DBUS) contains original header information before the L2 and L3 lookup and the result bus (RBUS) contains the results after both L3 and L2 lookups.
The F2 module has 12 forwarding engines per module. We need to determine which Clipper ASIC is the FE for port Eth6/4. We can use the following command to verify:
module-6# show hardware internal dev-port-map
--------------------------------------------------------------
CARD_TYPE: 48 port 10G
>Front Panel ports:48
--------------------------------------------------------------
Device name Dev role Abbr num_inst:
--------------------------------------------------------------
> Clipper FWD DEV_LAYER_2_LOOKUP L2LKP 12
+-----------------------------------------------------------------------+
+----------------+++FRONT PANEL PORT TO ASIC INSTANCE MAP+++------------+
+-----------------------------------------------------------------------+
FP port | PHYS | MAC_0 | L2LKP | L3LKP | QUEUE |SWICHF
...
3 0 0 0 0 0 0
4 0 0 0 0 0 0
From the output above, we can see that Eth6/4 is on Clipper (L2LKP) instance 0.
module-6# elam asic clipper instance 0
module-6(clipper-elam)# layer2
module-6(clipper-l2-elam)#
Configure the Trigger
The Clipper ASIC supports ELAM triggers for multiple frame types. The ELAM trigger must align to the frame type. If the frame is an IPv4 frame then the trigger must also be IPv4. An IPv4 frame will not be captured with an "other" trigger. The same logic applies to IPv6. You can see the different frame types supported by Clipper below:
module-6(clipper-l2-elam)# trigger dbus ?
arp ARP Frame Format
fc Fc hdr Frame Format
ipv4 IPV4 Frame Format
ipv6 IPV6 Frame Format
other L2 hdr Frame Format
pup PUP Frame Format
rarp Rarp hdr Frame Format
valid On valid packet
For NX-OS you can utilize the question mark to help parse out the ELAM trigger. You'll notice that there are several options available for F2 ELAM.
module-6(clipper-l2-elam)# trigger dbus ipv4 ingress if ?
<CR>
destination-ipv4-address destination ipv4 address
destination-mac-address Inner destination mac address
source-index Source index
source-ipv4-address source ipv4 address
source-mac-address Inner source mac address
vlan Vlan
etc…
For this example we want to capture the frame based off source and destination IPv4 address so we will only specify those values.
Clipper requires a trigger to be set for the DBUS and the RBUS. Different from M-series modules, there is no requirement to specify a packet buffer instance which helps simplify the RBUS trigger.
DBUS Trigger
module-6(clipper-l2-elam)# trigger dbus ipv4 ingress if source-ipv4-address 10.1.1.101 destination-ipv4-address 10.1.1.102
RBUS Trigger
module-6(clipper-l2-elam)# trigger rbus ingress if trig
Start the Capture
Now that the ingress FE has been selected and we've configured our trigger, we can start the capture
module-6(clipper-l2-elam)# start
We can check the status of the ELAM via the status command.
module-6(clipper-l2-elam)# status
ELAM instance 0: L2 DBUS Configuration: trigger dbus ipv4 ingress if source-ipv4-address 10.1.1.101 destination-ipv4-address 10.1.1.102
L2 DBUS Armed
ELAM instance 0: L2 RBUS Configuration: trigger rbus ingress if trig
L2 RBUS Armed
Once the frame matching the trigger has been received by the FE we will see the ELAM as triggered:
module-6(clipper-l2-elam)# status
ELAM instance 0: L2 DBUS Configuration: trigger dbus ipv4 ingress if source-ipv4-address 10.1.1.101 destination-ipv4-address 10.1.1.102
L2 DBUS Triggered
ELAM instance 0: L2 RBUS Configuration: trigger rbus ingress if trig
L2 RBUS Triggered
Interpret the Results
We can display the results via the show dbus and show rbus command. Below is an excerpt of the ELAM data that is most relevant in this example.
(some output omitted)
module-6(clipper-l2-elam)# show dbus
--------------------------------------------------------------------
L2 DBUS CONTENT - IPV4 PACKET
--------------------------------------------------------------------
...
vlan : 0xa destination-index : 0x0
source-index : 0x3 bundle-port : 0x0
sequence-number : 0x3f vl : 0x0
...
source-ipv4-address: 10.1.1.101
destination-ipv4-address: 10.1.1.102
destination-mac-address 0050.56a1.1aef
source-mac-address: 0050.56a1.1a01
module-6(clipper-l2-elam)# show rbus
--------------------------------------------------------------------
L2 RBUS INGRESS CONTENT
--------------------------------------------------------------------
l2-rbus-trigger : 0x1 sequence-number : 0x3f
di-ltl-index : 0x2 l3-multicast-di : 0x0
source-index : 0x3 vlan-id : 0xa
From the DBUS data above we can validate the frame was received on Vlan10 (vlan: 0xa) with a source MAC of 0050.56a1.1a01 and a destination MAC of 0050.56a1.1aef. We can also see that this is an IPv4 frame sourced from 10.1.1.101 destined to 10.1.1.102. There are several other fields not included in this output such as TOS value, IP flags, IP length, L2 frame length, etc... that are also often useful to check.
We can also validate what port the frame was received on via the source-index (the source LTL). For Nexus 7000, we can map an LTL to a port or group of ports via the following command:
N7K# show system internal pixm info ltl 0x3
Type LTL
---------------------------------
PHY_PORT Eth6/4
The above output shows that source-index of 0x3 maps to port Eth6/4. This confirms that the frame was received on Eth6/4.
From the RBUS data we can validate that the frame was switched on Vlan10 (vlan-id: 0xa). We can confirm the egress port from the di-ltl-index (destination LTL):
N7K# show system internal pixm info ltl 0x2
Type LTL
---------------------------------
PHY_PORT Eth6/3
The above output shows that the di-ltl-index of 0x2 maps to port Eth6/3. This confirms that the frame was switched out Eth6/3.
Further Information
Another command to remember is show system internal pixm info ltl-region, which will show how the switch has allocated the pool of LTL's. This is useful to understand the purpose of an LTL if it does not match to a physical port. A good example is a drop LTL:
N7K# show system internal pixm info ltl 0x11a0
0x11a0 is not configured
N7K# show system internal pixm info ltl-region
LTL POOL TYPE SIZE RANGE
=====================================================================
DCE/FC Pool 1024 0x0000 to 0x03ff
SUP Inband LTL 32 0x0400 to 0x041f
MD Flood LTL 1 0x0420
Central R/W 1 0x0421
UCAST Pool 1536 0x0422 to 0x0a21
PC Pool 1720 0x0a22 to 0x10d9
LC CPU Pool 32 0x1152 to 0x1171
EARL Pool 72 0x10da to 0x1121
SPAN Pool 48 0x1122 to 0x1151
UCAST VDC Use Pool 16 0x1172 to 0x1181
UCAST Generic Pool 30 0x1182 to 0x119f
LISP Pool 4 0x1198 to 0x119b
Invalid SI 1 0x119c to 0x119c
ESPAN SI 1 0x119d to 0x119d
Recirc SI 1 0x119e to 0x119e
Drop DI 2 0x119f to 0x11a0
UCAST (L3_SVI_SI) Region 31 0x11a1 to 0x11bf
UCAST (Fex/GPC/SVI-ES) 3648 0x11c0 to 0x1fff
UCAST Reserved for Future Use Region 2048 0x2000 to 0x27ff
======================> UCAST MCAST BOUNDARY <======================
VDC OMF Pool 32 0x2800 to 0x281f