ASR9000/XR: How to use Port Spanning or Port Mirroring - Page 3

xthuijs · ‎03-20-2011

Introduction

This document provides some extra documentation and use cases on the use of port spanning or port mirroring.

You can monitor traffic passing in & out of a set of L2 or L3 Ethernet interfaces (including bundle-Ether).

Core Issue

ASR 9000 is the only platform implementing SPAN on XR (Only support on ethernet linecards, not on SIP-700.)

You can use SPAN/Mirror in the follow scenarios

- L2 & L3 interfaces.
- Local, R-SPAN, and PW-SPAN only (no ER SPAN.)
- Scale limits:
    8 monitor sessions
    800 total source ports
    1.5 Gig bidirectional replication limit toward fabric for bundle interfaces and 10 Gig ports.
    Guideline: ~ 10% - 15% total bandwidth can be mirrored system-wide
- Source ports: Physical, EFPs, and bundles interfaces (L2 & L3)
- Destination ports: Ethernet interfaces, EFPs, and PW-SPAN. (No bundle) [ only L2 transport interfaces are supported as destination ports]

- Ability to use ACL's to define which traffic is to be captured

- Capture multicast traffic is possible

Note: some of the functionality mentioned are enhancements to the XR 4.0.1 release, this document assumes you are using this release or later.

A good reference on the terminology of SPAN/Mirror can be found here:

http://www.cisco.com/en/US/docs/switches/lan/catalyst6500/ios/12.2SX/configuration/guide/span.pdf

SPAN order of operation

SPAN mirrors what is on the wire
For ingress, this means packets are mirrored before QOS, ACL, and encapsulation rewrite operations.
For egress, this means packets are mirrored after QOS, ACL, and encapsulation rewrite operations.

Partial Packet Mirroring

User can configure to mirror first 64 upto 256 bytes of the packet.
Note: The actual mirrored packet will be the configured size plus 4-byte trailling CRC.

Sample config:

interface GigabitEthernet0/6/0/20 l2transport
monitor-session PW
mirror first 100 <== valid range: [64, 256], inclusively
!
!

Note: The mirrored packet received at sniffer will have the size of 104
(4-byte of trailing CRC added by transmit MAC layer.)

ACL based Mirroring

“permit/deny” determines the behavior of the regular traffic (forwarded or dropped)
“capture” determines whether the packet is mirrored to the SPAN destination.

On SPAN: mirror traffic on the wire (regardless with or without ACL.)

      ACL on ingress direction:
           SPAN will mirror traffic even regular traffic dropped by ACL: Always mirror!
     ACL on egress direction
          Will mirror if regular traffic is forwarded (Permit)
          Will not mirror if regular traffic is dropped (Deny.)

Inconsistent configurations:
“acl” is configured on SPAN source port but
   ACL has no “capture” keyword:
    No traffic gets mirrored.
“acl” is NOT configured on SPAN source port but
   ACL has “capture” keyword:
    Mirroring traffic as normal, no ACL performed.

The ACL can also be an L2 ACL :

ethernet-services access-list esacl_t2
10 deny 1234.5678.90ab 0000.0000.0000 any capture

L3 Spanning Example

monitor-session TEST
destination interface GigabitEthernet0/1/0/2 (<<<< this is NP3)
!
interface GigabitEthernet0/1/0/14 (<<<< this is NP2)
ipv4 address 5.5.1.1 255.255.255.0
monitor-session TEST
acl
!
load-interval 30
ipv4 access-group span ingress
!
ipv4 access-list span
10 permit ipv4 any host 1.1.1.10 capture
15 permit ipv4 any host 239.1.1.1 capture
20 permit ipv4 any host 2.2.2.100
30 permit ipv4 any any

Sample TRAFFIC GEN: (sending multicast in this example)
tgn rate 1000
L2-dest-addr 0100.5E01.0101
L2-src-addr 0003.A0FD.28A8
L3-src-addr 5.5.1.2
L3-dest-addr 239.1.1.1

Checking NP2: (the port that we are spanning)
Show global stats counters for NP2, revision v3

Read 12 non-zero NP counters:
Offset Counter                                         FrameValue   Rate (pps)
-------------------------------------------------------------------------------
22 PARSE_ENET_RECEIVE_CNT                                  5478        1001
31 PARSE_INGRESS_DROP_CNT                                     3           1
33 RESOLVE_INGRESS_DROP_CNT                                5474        1000
(there is no mcast recipient for this mcast addr, but we’re still replicating, see red line)
40 PARSE_INGRESS_PUNT_CNT                                     1           0
50 MODIFY_RX_SPAN_CNT                                      5475        1000
54 MODIFY_FRAMES_PADDED_CNT                                5475        1000
68 RESOLVE_INGRESS_L3_PUNT_CNT                                1           0
104 LOOP                                                       1           0
224 PUNT_STATISTICS                                            9           2
480 RESOLVE_IPM4_ING_RTE_DROP_CNT                           5475        1000
565 UIDB_TCAM_MISS_AGG_DROP                                    3           1
570 UIDB_TCAM_MISS_PORT4_DROP_FOR_HOST                         3           0

NP3 is the span monitor interface:
Show global stats counters for NP3, revision v3

Read 16 non-zero NP counters:
Offset Counter                                         FrameValue   Rate (pps)
-------------------------------------------------------------------------------
22 PARSE_ENET_RECEIVE_CNT                                    36           0
23 PARSE_FABRIC_RECEIVE_CNT                               79656        1000
30 MODIFY_ENET_TRANSMIT_CNT                               79655        1000

Packets received from fabric and sent off to the Ethernet on the span port!

PW SPAN example

For PW span to work, you need to define a local monitor session with a destination pseudo wire. You apply that span session to the interface of interest and define an xconnect group that also leverages that span session as one of the pw ends.

On the remote side where the PW terminates, you just configure regular VPWS.

Here an example:

On the Local Side, besides my Span configuration, there is also a local cross connect between the interested session we want to span over the PW

l2vpn

xconnect group TEST
p2p TEST
interface GigabitEthernet0/1/0/39

! port 39 is the port where we apply the span on.
interface GigabitEthernet0/1/0/20.100
! this is just a random AC to have traffic flowing between the spanned port.
!

AC configuration:

interface GigabitEthernet0/1/0/20.100 l2transport
encapsulation dot1q 100
rewrite ingress tag pop 1 symmetric
! the tag is popped because the other XCON end is a plain ethernet without vlan. The explanation and use cases of tag popping can be found a related

! Tech note article.

Configuration on the remote side:

Regular VPWS configuration:

RP/0/RSP0/CPU0:A9K-TOP#sh run l2vpn
l2vpn
xconnect group PW-SPAN
p2p PW-SPAN_1
   interface GigabitEthernet0/0/0/39
   neighbor 2.2.2.2 pw-id 1
   !
!
!
interface GigabitEthernet0/0/0/39
load-interval 30
transceiver permit pid all
l2transport
!
!

the neighbor in the l2vpn configuration is the LDP neighbor ID
between which the PW is built.

Show on remote side:
RP/0/RSP0/CPU0:A9K-TOP#show l2vpn xcon group PW-SPAN det

Group PW-SPAN, XC PW-SPAN_1, state is up; Interworking none
AC: GigabitEthernet0/0/0/39, state is up
    Type Ethernet
    MTU 1500; XC ID 0x4000a; interworking none
    Statistics:
      packets: received 0, sent 16570475
      bytes: received 0, sent 994228500

! packets received from the PW are sent out hte Attachment circuit's interface. The analyzer is connected to G0/0/0/39
PW: neighbor 2.2.2.2, PW ID 1000, state is up ( established )
    PW class not set, XC ID 0x4000a
    Encapsulation MPLS, protocol LDP
    PW type Ethernet, control word disabled, interworking none
    PW backup disable delay 0 sec
    Sequencing not set

      MPLS         Local                          Remote
      ------------ ------------------------------ -----------------------------
      Label        16002                          16027
      Group ID     0xa40                          0x2
      Interface    GigabitEthernet0/0/0/39        PW/TM/MS
      MTU          1500                           1500
      Control word disabled                       disabled
      PW type      Ethernet                       Ethernet
      VCCV CV type 0x2                            0x2
                   (LSP ping verification)        (LSP ping verification)
      VCCV CC type 0x6                            0x6
                   (router alert label)           (router alert label)
                   (TTL expiry)                   (TTL expiry)
      ------------ ------------------------------ -----------------------------
    MIB cpwVcIndex: 4294705162
    Create time: 04/04/2011 14:36:42 (00:20:07 ago)
    Last time status changed: 04/04/2011 14:36:42 (00:20:07 ago)
    Statistics:
      packets: received 16570475, sent 0
      bytes: received 994228500, sent 0

! Packets received on the Pseudo Wire from the SPAN port

NOTE: Pseudo Wire counters on the span side are not incrementing.That is the XCON group "cisco" in this picture config example.

This is intentional. You can review the SPANNING also with this command:

RP/0/RSP1/CPU0:A9K-BOTTOM#sh monitor-session counters

Monitor-session PW_TM_MS
GigabitEthernet0/1/0/39
    Rx replicated: 58488205 packets, 3743245120 octets
    Tx replicated: 58488206 packets, 3743245184 octets
    Non-replicated: 0 packets, 0 octets

R-SPAN configuration:

R-SPAN is natively support with the capability of ASR9000 to do vlan imposition:

monitor-session MS2

destination interface gig0/2/0/19.10

!

interface gig0/2/0/12.10 l2transport

encapsulation dot1q 10 <<< Monitoring vlan 10 traffic

monitor-session MS2

!

interface gig0/2/0/19.10 l2transport (*)

encapsulation dot1q 100 <<< VLAN 100 will get imposed.

!

(*) Monitor destination could be any supported destination interface regardless of monitor source

Related Information

n/a

Xander Thuijs, CCIE #6775

Sr. Tech Lead ASR9000

xthuijs · ‎07-06-2015

btw I saw from your earlier note which had a few links to the counters that you have a trident card it seems (since I see IFDMA discards). this means that you are indeed hitting a pps limitation.

I also see in the NP TM counters that there are a lot of WRED drops in the TM, this is caused by a BW limitation of the replicator.

so yeah in your case you are hitting a forwarding capability problem on the TM that receives the traffic that it needs to replicate.

regards

xander

douglas.vujanic · ‎07-14-2015

I seem to not be able to SPAN a port that I know is tagged and have the captured packets be tagged accordingly. my config is...

int gi101/0/0/01.10

ipv4 address 10.10.10.1 255.255.255.252
monitor-session test ethernet
mirror first 100
!
encapsulation dot1q 10

int gi0/0/0/2 l2transport

am I missing something?! does SPAN off a 9000nv satellite port make a difference? incidentally, xr 5.1.3

Aleksandar Vidakovic · ‎07-15-2015

There are restrictions on using satellite interface as SPAN destination, but in your case you're using it as source.

You could check the 3 points that Xander wrote on another thread in this doc:

show controller np counter np <of spanned interface> and see if the packet rate of ENET_RECEIVE is showing similar counts for MDF_SPAN
show controller np tm counters to see if the traffic manager is hitting a bw limitation for the span operation.
are there any early fast discards on the npu counters that signify that the npu is running out of steam.

/Aleksandar

douglas.vujanic · ‎07-15-2015

I'm seeing some traffic at the destination but it's not being received as tagged and I know it is tagged (I have a physical TAP on the wire simultaneously). This is where my concern lies. I want the destination to not alter the frames and I want to receive the frames on my sniffer tagged. Is my config correct or what should I alter??

Aleksandar Vidakovic · ‎07-20-2015

Hi Douglas,

the asr9k should send the frame to the SPAN destination with the dot1q tag included. I have verified this in the lab. If you are testing this with a Typhoon line card and XR version 4.3.x or later, you can use the "monitor np counter" feature to confirm this. Monitor the PARSE_SPAN_LPBK counter. See https://supportforums.cisco.com/document/122386/asr9000xr-how-capture-dropped-or-lost-packets for more on this feature.

It's possible that the SPAN destination device strips off the dot1q tag. It happened to me when i was using laptop as a sniffer.

hth,

Aleksandar

ahmed zaidi · ‎10-15-2015

Hi Xander,

i would like to know, how can I configure this monitor session on cisco ASR 9010?

monitor configuration on cisco 7600:

monitor session 1 source interface Gi1/2 , Gi1/4 - 7 , Gi1/12 - 13 , Gi1/18
monitor session 1 source interface Gi2/2 , Gi2/16 , Gi2/20

Many thanks,

Br,

Ahmed

xthuijs · ‎10-15-2015

first define a template for the destination interface:

monitor-session TEST
destination interface GigabitEthernet0/1/0/2

Then the span sources individually need to be configured like this:

interface GigabitEthernet0/1/0/14
monitor-session TEST

so "session 1" in your 7600 example equates to "TEST" here

and source interface x/y/z is the equivalent of appling monitor-session TEST to the respective interface you want to capture from.

regards

xander

ahmed zaidi · ‎10-15-2015

Thanks Xander,

i have another question can i apply more than one monitor session in the same interface ?

Many thanks,

Br,

Ahmed

xthuijs · ‎10-15-2015

now that you can't. it is one session only.

xander

ahmed zaidi · ‎10-16-2015

ok, many Thanks,

Br,

Ahmed

kjclark · ‎12-10-2015

Hello Xander,

I work with a large global provider and our asr9ks are used in the core running mpls and isis. We have a scenario in which our customer's traffic is being "dropped" (retransmissions in the trace) somewhere in the network (trace taken between PE/PE).

We have some asr1k (makes it easy to capture traffic) and some 7600 (not so easy), and 1-3 asr9k in the path (depending on which path the customer traffic is taking and during testing when we move the traffic to better isolate the path and insure that a particular link isn't causing the problem).

I would like to know if it is possible (and how) to capture a single customer specific (vrf) tcp flow through the mpls cloud?

As there is also no example here, I am thinking it is not possible through the asr9k as the traffic is obviously lable switched, but I defer to your expertise and look forward to some solution aside from inserting a switch between our core nodes.

Brgds,

Kevin

xthuijs · ‎12-11-2015

hi kevin,

you can capture labeled traffic that is not the problem per-se, but capturing specific labeled traffic is more of a difficulty. simply because we can't match on a particular label value.

we do have some possibilities to match an ip ACL on labeled traffic if it is mpls/ip which is what l3vpn is. if it is l2vpn, we'd be stuck since there is an ether header in the middle then.

so you could try to enable an l3 acl on a core interface matching the traffic and see if it can be spanned like this. this ip acl on labeled is a rather new func from I want to say 524, but whether it works in conjunction with SPAN that I have never tried, but worth the shot possibly.

alternatively, you'd be probably stuck spanning it all, and sift through the capture.

or if you have that luxury, configure a parallel path to be used only for that circuit so it is easier to capture just that traffic, but it is not really a true test anymore since you are making a new path, but possibly something worth to consider.

retransmissions generally mean a packet loss. looking at the path to identify if there are drops and for what reasons, which can be done quite easily might give some pointers also such as label errors, punt adj, lpts policer drops, npu overload (efd discards) etc.

cheers!

xander

kjclark · ‎12-11-2015

Hi Xander,

First, thanks for getting back to me so quickly, I have been working with Serge K and Brian D on the issue.

We are running 5.1.3 on our asr9k, and accordingly cannot config for l3vpn.

However, we may have a solution to forward the traffic to a capture point (based on PW) elsewhere in the network.

Again, Thanks vm for your suggestions/support.

Have a great weekend.

Brgds,

K

xthuijs · ‎12-12-2015

ah sounds good Kevin, yup you're in good hands with them! a pw span is an option also indeed!

regards!

xander

braintrust-nward · ‎12-20-2015

Hi Xander,

I have an ASR9001 configured to mirror BNG subscribers (PPP and IPoE) to a PW. This PW goes to an ME3600X where we have a monitoring PC connected for debugging customers.

PW is up, monitor session counters shows packets, but the PW does not show any packets.

Subscribers are on bundle-ether.

I have a service defined and attach the service with CoA (test radius coa etc.), and the service shows against the customer session.

Are there restrictions with this configuration with bundle-ether subscribers, or something?

Relevant config:

monitor-session MS-CustomerDebug ethernet
destination pseudowire
!
l2vpn
pw-class Mirror
encapsulation mpls
transport-mode ethernet
!
!
xconnect group CustomerMirror
p2p CustomerMirror
monitor-session MS-CustomerDebug
neighbor ipv4 103.241.56.10 pw-id 2022303
pw-class Mirror
!
!
!
!
dynamic-template
type ipsubscriber MIRROR
monitor-session MS-CustomerDebug ethernet
!
!
!