Cisco Support Community
cancel
Showing results for 
Search instead for 
Did you mean: 

ASR9000/XR: Multichassis LAG or MC-LAG (MCLAG) guide

 

Introduction

Multichassis LAG is a tricky concept. In general the members of a bundle (also called LAG, Link Aggregation Group, Etherchannel, Portchannel) are between 2 distinct devices. The advantage of using a bundle is that there is a single routing peering, no worries about spanning tree and things like that. However the redundancy is compromised when either one of the peers fail. Using ECMP (Equal Cost Multipath) in L3 scenarios allows me to dual home to 2 different devices so I have a back up also when one of the peers fail for me, but that negates the benefit of using bundle having a single routing peering.

MC-LAG attempts to provide a means to allow me to dual home a device (DHD, the dual homed device) to two different peer devices (the POA, or Point of Attachment), so basically allowing me to have the benefits of node redundancy, while maintaining single peerings which makes my L2 (Spanning Tree/ STP) or L3 (no dual peerings) life a lot easier.

Does it come with restrictions? Of course! It's technology, nothing comes for free...! So in this document we will highlight how to set it up, what the restrictions are that you need to be aware of and how to troubleshoot and verify MC-LAG scenarios.

 

Overview

MC-LAG & ICCP enable a switch/router to use standard Ethernet Link Aggregation for device dual-homing, with active/standby redundancy

Dual-homed Device (DHD) operates as if it is connected to single virtual device and runs IEEE std. 802.1AX-2008 (LACP)

Point of Attachment (PoA) nodes run Inter-chassis Communication Protocol (ICCP) to synchronize state & form a Redundancy Group (RG)

 

Screen Shot 2013-07-15 at 8.36.30 AM.png

Idea is to let the peer “device” feel that it’s connected to a single “device” •à need information sync between two PoA.

MC-LAG uses ICCP to synchronize LACP configuration & operational state between PoAs, to provide DHD the perception of being connected to a single switch. All PoAs use the same System MAC Address & System Priority when communicating with DHD

Configurable or automatically synchronized via ICCP

Every PoA in the RG is configured with a unique Node ID (value 0 to 7). Node ID + 8 forms the most significant nibble of the Port Number.
For a given bundle, all links on the same PoA must have the same Port Priority.
 

1. Port Status

Currently all vendor’s MC-LAG implementation is based on active/standby mode. All the member ports on standby POA must be in standby mode
which doesn’t forward any packet.
The POA need to be configured with higher system priority (low value) than DHD. So POA determine which link is in active/standby state
By default, all member ports on the active POA is in active state unless:
– If active POA has “maximum active-links” configured, then it only put that amount of ports into active
– If DHD has “maximum active-links” configured, then only that amount of ports can be put into active
Active POA determine which ports goes to active or standby based on the port priority and port number on its local port as normal LACP
Failure events can cause bundle member port active/standby state failover
 
 
Coupled vs. Decoupled

MC-LAG bundle (sub-)interface can be configured for both L2 and L3 service

 

Service redundancy status may or may not be tied to PoA/bundle active/standby status

 

P2P PW (coupled mode): bundle state determine the PW state. If bundle is in active state, then it advertise “active” PW status message.

 

Otherwise it will advertise “standby” PW status message to its peer Routers

 

H-VPLS access P2P PW (coupled & one-way mode): PW and its backup PW are in regular “one-way” PW redundancy mode  on active POA.

 

On the standby POA, both of itse PWs are in standby state

 

VPLS service (de-coupled mode): regardless if bundle is active or standby, VPLS PWs are always in active forwarding state

 

H-VPLS access PW (PW under bridge-domain): same de-coupled mode as VPLS

 

L3 service (coupled mode): bundle state determine the L3 sub-interface state.

 

If bundle is in active state, then bundle L3 interface/sub-interface keep up. Otherwise, it keeps in protocol “down” state

Screen Shot 2013-07-15 at 8.43.33 AM.png
This example above is for p2p two-way PW redundancy. Both sides must run MC-LAG. It’s in coupled mode. Bundle state decide PW
redundancy state. Active POA send active PW status to remote Router. Standby POA send standby PW status. PW become active ONLY if
local and remote Routers are both active. The rest of 3 PWs are in standby mode. When active link fail, the PoA and bundle will switchover.
Then it will advertise new active or standby PW status message to the peer device, which cause PW switchover as well
Screen Shot 2013-07-15 at 8.44.03 AM.png
This example is the VPLS service.
The remote Routers don’t have to run MC-LAG. The remote side could be singe or dual home Router, could be vpls or spoke PW
VPLS PW status is independent of the link bundle or POA status. They are all in active state unless remote Router signal a standby status. This is called decouple mode. ASR9K only support decouple mode for VPLS PW and bridge-domain access PW
Note, 7600 support both couple and decouple mode. And coupled mode is on by default. In coupled mode, the Bundle AC state will decide the
PW state. With coupled mode, if POA is in standby mode, then its VPLS PW will be in standby mode as well
POA (bundle) failover will trigger LDP MAC withdrawal message to remote Routers
 
 
 
 

Configuration

 

The following configuration is recommended for mLACP operation. 

 

2. ICCP setup (POAs)

 

First,  an ICCP group must be set up. This configuration is not owned by the  bundle infra, so only a simple example setup is provided here. For more  details please refer to the information provided by the ICCP team.

On each POA, the following is required:

  • An IPv4 address at each end of the link between the POAs to provide IP connectivity.
  • An IP address set up on a loopback interface as the endpoint for the LDP link over which ICCP operates;
  • A  routing protocol so that the routes can be determined (static route in  the example, for which proxy-arp on the connection between POAs is  required);
  • An LDP session to connect to the peer.

 

RP/0/0/CPU0:ios(config)#interface <link connected to peer POA>
RP/0/0/CPU0:ios(config-if)#ipv4 address <IP address; e.g. 4.0.0.[1-2]/8>
RP/0/0/CPU0:ios(config-if)#proxy-arp
RP/0/0/CPU0:ios(config-if)#root
RP/0/0/CPU0:ios(config)#interface Loopback <ID> ipv4 address <IP address; e.g. 5.4.3.[1-2]/32>
RP/0/0/CPU0:ios(config)#router static address-family ipv4 unicast <IP of peer POA, e.g. 5.4.3.[2-1]/32> <IP of far end of link between POAs, e.g. 4.0.0.[2-1]>
RP/0/0/CPU0:ios(config)#mpls ldp
RP/0/0/CPU0:ios(config-ldp)#router-id <unique ID, e.g. 5.4.3.<1-2>>
RP/0/0/CPU0:ios(config-ldp)#discovery targeted-hello accept
RP/0/0/CPU0:ios(config-ldp)#log neighbor
RP/0/0/CPU0:ios(config-ldp)#interface <link connected to other POA>
RP/0/0/CPU0:ios(config-ldp)#commit

The next step is to add an ICCP redundancy group on each POA; using the same group ID for both:

RP/0/0/CPU0:poa1(config)#redundancy iccp group <group-id> member neighbor <LDP router-id of other POA>
RP/0/0/CPU0:poa1(config)#commit

NB: The same ICCP group ID must be used on both POAs.

At  this point the two POAs should connect and start running ICCP. However,  it may take a little while for the ICCP connection to be established.  Once connected, the peer POA's information should appear in the show iccp group <group-id> command, and its state should be up (connected). NB:  The configuration and commands up to this point are not owned by the  bundle infra; please refer to the information from the ICCP team for  more details. (However, if you investigate and find these instructions  incorrect or out of date, please let us know!)

 

3. mLACP setup (POAs)

 

The  underlying ICCP session has now been established. To enable the mLACP  session, some further configuration settings are required under the ICCP  group:

  • The system MAC, which should be the same on both POAs;
  • The system priority, which should be the same on both POAs, and a priority of 1 is recommended;
  • The node ID, which must be different on each POA.

 

RP/0/0/CPU0:ios(config)#redundancy iccp group <ICCP group ID>
RP/0/0/CPU0:ios(config-redundancy-iccp-group)#mlacp system mac <ID unique to group, same on both POAs>
RP/0/0/CPU0:ios(config-redundancy-iccp-group)#mlacp system priority 1
RP/0/0/CPU0:ios(config-redundancy-iccp-group)#mlacp node <ID unique to each device in the group>
RP/0/0/CPU0:ios(config-redundancy-iccp-group)#commit

Next,  a bundle needs to be added to the ICCP group, with a configured MAC  address. In order to protect against flaps during switchover operations,  please add the recommended configuration items below. Matching  configuration is required on both POAs. This includes using the same  number for the bundle interface, and the same MAC address.

RP/0/0/CPU0:ios(config)#interface Bundle-Ether <x>
RP/0/0/CPU0:ios(config-if)#mac-address 0001.0002.0003
RP/0/0/CPU0:ios(config-if)#bundle wait-while 100
RP/0/0/CPU0:ios(config-if)#lacp switchover suppress-flaps 300
RP/0/0/CPU0:ios(config-if)#mlacp iccp-group <ICCP group ID>
RP/0/0/CPU0:ios(config-if)#commit

On the POAs, the last step is to add members to the bundle, as you would with normal LACP. The period configuration is optional; it enables faster LACP timeouts.

RP/0/0/CPU0:ios(config)#interface <Ethernet interface>
RP/0/0/CPU0:ios(config-if)#bundle id <x> mode active
RP/0/0/CPU0:ios(config-if)#lacp period short

 

4. DHD setup

 

All  that's left is to add the bundle and member configuration on the DHD.  If the DHD is an XR box running the new software, the configuration is  as follows:

RP/0/0/CPU0:ios(config)#interface Bundle-Ether <y>
RP/0/0/CPU0:ios(config-if)#bundle wait-while 100
RP/0/0/CPU0:ios(config-if)#lacp switchover suppress-flaps 300
RP/0/0/CPU0:ios(config-if)#root
RP/0/0/CPU0:ios(config)#interface <each interface connected to POAs>
RP/0/0/CPU0:ios(config-if)#bundle id <y> mode active
RP/0/0/CPU0:ios(config-if)#lacp period short
RP/0/0/CPU0:ios(config-if)#commit

Any  non-XR device (or XR up to R3.9) should have a bundle and members  configured similarly. NB: Against some implementations the expected  behavior is that the bundle will flap on a switchback event for up to 2  seconds. To avoid this, lacp fast-switchover (IOS) or equivalent configuration is required.

If lacp fast-switchover or similar configuration is not available on the DHD, lacp switchover suppress-flaps 2500 configuration can be added on the bundle on the POAs to avoid the state  flap. However, this will result in ~2 seconds of traffic loss on the  switchback event (though the bundle stays up).

If the lacp switchover suppress-flaps configuration or some kind of state dampening is not used or not  available on the DHD, a bundle flap on the DHD on a switchover or  switchback event is expected.

 

5. Checking status

 

The  members added to the bundle on one POA will go Active, and the members  on the other POA will be in Standby state. This can be verified using show bundle on either POA to display the membership information for correctly configured members on both the POAs:

RP/0/0/CPU0:ios#show bundle 
Mon Jun  7 06:02:46.507 PDT

Bundle-Ether1
  Status:                                    Up
  Local links <active/standby/configured>:   1 / 0 / 1
  Local bandwidth <effective/available>:     1000000 (1000000) kbps
  MAC address (source):                      0000.deaf.0000 (Configured)
  Minimum active links / bandwidth:          1 / 1 kbps
  Maximum active links:                      64
  Wait while timer:                          100 ms
  Load balancing:                            Default
  LACP:                                      Operational
    Flap suppression timer:                  300 ms
    Cisco extensions:                        Disabled
  mLACP:                                     Operational
    ICCP Group:                              1
    Role:                                    Active
    Foreign links <active/configured>:       0 / 1
    Switchover type:                         Non-revertive
    Recovery delay:                          300 s
    Maximize threshold:                      Not configured
  IPv4 BFD:                                  Not configured

  Port                  Device           State        Port ID         B/W, kbps
  --------------------  ---------------  -----------  --------------  ----------
  Gi0/0/0/0             Local            Active       0x8001, 0x9001     1000000
      Link is Active
  Gi0/0/0/0             5.4.3.2          Standby      0x8002, 0xa001     1000000
      Link is marked as Standby by mLACP peer
RP/0/0/CPU0:ios#


Switchover

 


 

To switch which POA is active you can use the following CLI command on the currently active router:

mlacp switchover Bundle-Ether 1

The following example illustrates a switchover using this command.

RP/0/0/CPU0:ios#show bundle 
Mon Jun  7 06:02:46.507 PDT

Bundle-Ether1
  Status:                                    Up
  Local links <active/standby/configured>:   1 / 0 / 1
  Local bandwidth <effective/available>:     1000000 (1000000) kbps
  MAC address (source):                      0000.deaf.0000 (Configured)
  Minimum active links / bandwidth:          1 / 1 kbps
  Maximum active links:                      64
  Wait while timer:                          100 ms
  Load balancing:                            Default
  LACP:                                      Operational
    Flap suppression timer:                  300 ms
    Cisco extensions:                        Disabled
  mLACP:                                     Operational
    ICCP Group:                              1
    Role:                                    Active
    Foreign links <active/configured>:       0 / 1
    Switchover type:                         Non-revertive
    Recovery delay:                          300 s
    Maximize threshold:                      Not configured
  IPv4 BFD:                                  Not configured

  Port                  Device           State        Port ID         B/W, kbps
  --------------------  ---------------  -----------  --------------  ----------
  Gi0/0/0/0             Local            Active       0x8001, 0x9001     1000000
      Link is Active
  Gi0/0/0/0             5.4.3.2          Standby      0x8002, 0xa001     1000000
      Link is marked as Standby by mLACP peer
RP/0/0/CPU0:ios#mlacp switchover Bundle-Ether 1
Sun Jan 31 23:46:43.706 PST
This will trigger the peer device (0/0/CPU0 (0x0)) to become active for Bundle-Ether1. This may result in packet loss on the specified bundle.

Proceed with switch over? [confirm]

RP/0/0/CPU0:Jan 31 23:46:44.666 : BM-DISTRIB[282]: %L2-BM-5-MLACP_BUNDLE_ACTIVE : This device is no longer the active device for Bundle-Ether1
RP/0/0/CPU0:Jan 31 23:46:44.668 : BM-DISTRIB[282]: %L2-BM-6-ACTIVE : Gi0/0/0/0 (0x20000020) is no longer Active as part of Bundle-Ether1 (Not enough links available to meet minimum-active threshold)

RP/0/0/CPU0:ios#show bundle 
Mon Jun  7 06:04:17.778 PDT

Bundle-Ether1
  Status:                                    mLACP hot standby
  Local links <active/standby/configured>:   0 / 1 / 1
  Local bandwidth <effective/available>:     0 (0) kbps
  MAC address (source):                      0000.deaf.0000 (Configured)
  Minimum active links / bandwidth:          1 / 1 kbps
  Maximum active links:                      64
  Wait while timer:                          100 ms
  Load balancing:                            Default
  LACP:                                      Operational
    Flap suppression timer:                  300 ms
    Cisco extensions:                        Disabled
  mLACP:                                     Operational
    ICCP Group:                              1
    Role:                                    Standby
    Foreign links <active/configured>:       1 / 1
    Switchover type:                         Non-revertive
    Recovery delay:                          300 s
    Maximize threshold:                      Not configured
  IPv4 BFD:                                  Not configured

  Port                  Device           State        Port ID         B/W, kbps
  --------------------  ---------------  -----------  --------------  ----------
  Gi0/0/0/0             Local            Standby      0x8003, 0x9001     1000000
      mLACP peer is active
  Gi0/0/0/0             5.4.3.2          Active       0x8002, 0xa001     1000000
      Link is Active
RP/0/0/CPU0:ios#

This  command should cover most of the required information. However, for a  more detailed look, there is also an mLACP-specific command available.  This gives you information about each router in each redundancy group,  and the state and configuration each has advertized for each bundle.

 

RP/0/0/CPU0:ios#show mlacp 
Mon Jun  7 06:05:36.901 PDT

ICCP Group 1
  Connect timer:  Off

  Node  LDP ID           State         System ID                 Sync   Vers
  ----  ---------------  ------------  ------------------------  -----  ----
     1  Local            Up            0x0001,00-0d-00-0e-00-0f  Done   -   
     2  5.4.3.2          Up            0x0001,00-0d-00-0e-00-0f  Done   1   


  Bundle-Ether1 (ROID: 0000.0001.0000.0000)
    Node  Aggregator Name       State       Agg ID  MAC Address
    ----  --------------------  ----------  ------  ---------------
       1  Bundle-Ether1         Up          0x0001  0000.deaf.0000
       2  BE1                   Up          0x0001  0000.deaf.0000

RP/0/0/CPU0:ios#

 

Troubleshooting

In this section some basic verification tips that can be checked out that hopefully pinpoint to the culprit when MC-LAG is not working as we expected.

 

6. Information to Collect

Please collect the following information to be provided in any triage request for mLACP issues. Before submiting a TAC case please follow the steps set out in the following sections to diagnose some more common faults and misconfigurations: 

 

  1. Details of the symptoms, and which POA is experiencing them (with timestamps if possible). 
  2. The sequence of events that triggered the issue. [mLACP states can be history-dependent so it can be important to know the previous bundle configuration.] 
  3. The following logs from both of the POAs: 

    1. show tech bundle 

    2. show tech bundle in admin mode. 

    3. show iccp group 

    4. show iccp counters 

    5. show iccp trace 

  4. The following logs from the DHD: 
    1. show tech bundle 

    2. show tech bundle in admin mode.  

 

7. Is this a bundle issue?

Checking ICCP

If you are seeing unexpected mLACP behavior, the first thing to check the health of the ICCP layer (which mLACP relies on) to ensure it is in the expected state. Normally ICCP should be up with a group member present (unless you are testing split brain or device-level failure):

 

RP/0/0/CPU0:ios#show iccp group
Wed Feb 10 22:58:58.845 PST
Redundancy Group 1
  member ip:5.4.3.2 (ios), up (connected)
    monitor: route-watch (up)
  No backbone interfaces.
  enabled applications: mLACP 
RP/0/0/CPU0:ios#

 

If you have a backbone interface configured, it should normally be showing as up unless core isolation is in effect:

 

RP/0/0/CPU0:ios#show iccp group
Wed Feb 10 23:04:03.406 PST
Redundancy Group 1
  member ip:5.4.3.2 (ios), up (connected)
    monitor: route-watch (up)
  backbone interface Gi0/0/0/3: up
  enabled applications: mLACP 
RP/0/0/CPU0:ios#

 

If the output of this command is not what you are expect, the issue needs to be dealt with by the ICCP team. Please refer to information from the ICCP team, or contact them if you believe there is a problem.

 

Features & Protocols

 

If there is a problem with features that run on the bundle but the bundle IM state/LACP state is correct, then the team for the particular feature should be contacted. As with normal bundle interfaces, the Bundle Infra controls the bundle interface itself, and other components are responsible for running their services over the bundle.

This info will help you direct the TAC where to send the case to. Is it really an MC-LAG issue, or a feature issue (that might exist on a regular bundle also) or is there just an issue with teh bundle itself. These items are handled by different development groups, so it helps to drill down the failure issue to a most common set of problems that will inherently mean a faster case resolution as the sw problem can be dealt with immediatley with the right group.

 

8. Bundle Infra issues

The bundle is down

 

To identify the issue, please go through the following steps:

 

  1. Check whether ICCP is healthy, as described above. If not, this is likely to be the cause of the issue. Please correct this and see if the problem persists.
  2. Verify that the configuration is correct for the ICCP group and the bundles. Correct any misconfigurations and see if the issue persists.

  3. Check that the minimum-active links/bandwidth configuration is set to an appropriate value (i.e. smaller than the number of links in the bundle/their total bandwidth).

  4. The output of the show bundle command should tell you the reason any member links are not Active. The output of this command (combined with the table of reasons with additional information should indicate the cause of the problem and how to fix it.

  5. If you are still seeing that the bundle is down, please collect the information listed above.

 

 

9. The bundle flapped on the POA

 

Bundle flaps are sometimes expected on mLACP events, but they are usually the result of misconfigurations. So the first thing to do is check that all your configuration is correct. Please refer to the configuration guidance for details, and specifically take note of the following:

 

  1. If a protocol flap is seen (e.g. OSPF) but not a flap in the state of the bundle interface itself, please refer to the feature-specific wiki and follow up with the team responsible for the feature.
  2. The mlacp system priority and mlacp system mac configured under redundancy iccp group <x> must be configured to be the same value on both devices in the RG.

  3. The same mac-address should  be configured on the bundle interface on both POAs.

  4. If the DHD supports bundle wait-while 100 (XR 4.0) or lacp fast-switchover (XR up to 3.9 & IOS) or equivalent, this should be configured on the DHD.

  5. If the DHD has bundle wait-while 100 configuration, lacp switchover suppress-flaps 300 should be configured on the POAs. Otherwise, lacp switchover suppress-flaps 2500 is required.

  6. The bundle should be configured with bundle wait-while 100 on the POAs.

  7. Check for any syslogs or bistate alarms that could indicate the cause of the flap.

 

If you're still seeing a flap, please collect the information requested above.

 

 

10. The bundle flapped on the DHD

 

Bundle flaps on the DHD are usually expected during switchover events. If an XR 4.0 device is in use, you can configure bundle wait-while 100 and lacp switchover suppress-flaps 300 on the bundle to avoid a flap (assuming the POAs are set up correctly as above). If other DHD software provides similar functionality it can be used, otherwise a flap cannot be avoided.

 

 

11. Switchover did not occur

 

This is likely to mean one of the following:

 

  1. The bundle is Down on the Standby POA (and was Down before the switchover attempt). The output of show bundle on the Standby POA should indicate the reason for this.

  2. The two POAs are not communicating over ICCP. See the "Checking ICCP" section above for details.

 

12. Both POAs are Active

 

This should only happen when the POAs are failing to communicate over ICCP. See the "Checking ICCP" section above for details.

 

13. The bundle is Down on the Active POA

 

This can happen if:

 

  1. There is no Standby POA to switch over to, or the bundle is Down on the Standby POA.
  2. The required number of links have been put in LACP Selected state, but have not been Activated (e.g. due to failing to progress further in LACP negotiations). A switchover cannot be performed in this scenario. This could indicate a misconfiguration on the DHD.

 

14. The bundle is Up on the Standby POA

 

This is expected. Please see the section on Interface State in the events and scenarios section.

 

Events and Scenarios

15. Initial Bringup

 

By  default, when the first device is added to the ICCP group, it does not  enable any bundles in that ICCP group until negotiations with a peer  have been completed. This is to reduce churn when adding a device to the  group (or reloading the device).

This means that without an mLACP peer, mLACP bundles cannot be enabled. It could also lead to the following:

  1. Two routers (A and B ) are configured as mLACP peers and start operating their mLACP bundles.
  2. Router B crashes. Router A takes over operation of all bundles and continues running on its own.
  3. Router  A is reloaded. After it comes up it waits indefinitely for the peer,  and therefore does not bring up any of the mLACP bundles.

If this is undesirable, the following configuration can be used:

RP/0/0/CPU0:ios(config)#redundancy iccp group 1 mlacp connect timeout ?
  <0-65534>  Number of seconds to wait before assuming mLACP peer is down.
RP/0/0/CPU0:ios(config)#

If  a connect timeout is configured and no peer device is present, mLACP  bundles will be enabled once the timeout has elapsed after joining the  ICCP group.

 

16. mLACP Active and Standby

 

Each  of the mLACP peer devices is either the Active or the Standby device  for each bundle. On each device, its status (Active or Standby) is  displayed in the mLACP section of the show bundle output.

 

Interface State

 

The bundle interface state (as displayed in show interfaces)  can be Up or Down on either the Active or the Standby device for the  bundle. On both devices, under normal conditions, the bundle should be  Up.

The bundle interface state will be Down on the Active device if:

  1. The  minimum link threshold could not be met (or no links could be brought  up) and there is no mLACP peer advertising that it is ready to bring the  bundle Up (so a switchover cannot be performed).
  2. The  required number of links have been put in LACP Selected state, but have  not been Activated (e.g. due to failing to progress further in LACP  negotiations). Again, a switchover cannot be performed in this case.  This could indicate a misconfiguration on the DHD.

The bundle interface will be Down on the Standby device if:

  1. The device is not ready to forward bundle traffic; if the bundle goes down on the Active device, traffic will be lost.
  2. The device is cold standby for the bundle (see next subsection).

 

Hot vs Cold Standby

 

In the show bundle output, the state of the bundle on the standby POA could be either of the following:

  • mLACP Hot Standby:  This bundle has enough links available, it is Up in IM, and it has the  required POA configs to take over without a flap if the active router  goes down. NB: This does not guarantee that the bundle will take over without a flap. Incorrect/incompatible configuration on the DHD could still lead to a flap.

  • mLACP Cold Standby:  The bundle has enough links available to take over if required.  However, it is marked as Down in IM because it will have to be Down  initially after the switchover while LACP negotiations are in progress.  This is due to some missing configuration that is required for a  seamless switchover; e.g. lacp switchover suppress-flaps.

Please see the

 

17. Switchover

 

The  following subsections provide more details on mLACP switchover and  switchback events and the behavior under various conditions.

 

Switchover Types

 

The mLACP switchover method can be modified using the following CLI:

RP/0/0/CPU0:ios(config)#interface bundle-ether 1 mlacp switchover type ?
  brute-force    Force switchover by disabling all local member links
  revertive      Revert based on configured priority values
RP/0/0/CPU0:ios(config)#interface bundle-ether 1 mlacp switchover type 

These  options (and the default, when this configuration is not present) are  mutually exclusive, and the value must match on the bundle on both POAs.   They determine whether the dynamic priority management or brute force mechanism is used, and whether the behavior is revertive or non-revertive.

The behavior can also be modified to maximize the links or bandwidth available using one of the following CLIs:

interface <bundle> mlacp switchover maximize bandwidth [threshold <kbps>]
interface <bundle> mlacp switchover maximize links [threshold <count>]

This  causes the active device to be the one with more bandwidth or more  links available in the bundle. If a threshold is specified, this  behavior only takes effect once the available links/bandwidth on one  device falls below the threshold. This must be configured symmetrically  on both devices.

 

Dynamic Priority Management

 

Dynamic priority management means that a switchover is achieved gracefully by modifying the LACP port priorities.

  • Before any switchovers have occurred, the LACP port priorities for the members on each POA match the value configured using the mlacp port-priority configuration for the bundle interface, or 32768 as a default if there is no value configured.
  • To  perform a switchover, the priorities at which the links are operating  on at least one POA are changed automatically by the POA software so  that the links to the POA which is to go active are higher-priority  (have a smaller LACP priority number) than the links attached to the  other POA. (This means the configured value can differ from the  operational value.)
  • LACP  demands the highest-priority links must be selected ahead of  lower-priority ones (by the DHD as well as the POAs), which is why this  action triggers a switchover.
  • In  almost all cases, the port priority of the currently-active POA is  reduced (the operational priority number is increased) to trigger a  switchover to the peer POA. This avoids churn in the LACP state machines  on the links to the POA becoming active.
  • If  this is not possible (i.e. if the operating priority of the standby POA  is 65535 so there are no larger numbers available) the priorities of  both are modified; the newly active POA's links take priority 1 and the  other POA's take priority 2.

It  is possible to reset the priorities used on both POAs to their initial  values. Doing so will cause a switchover if the device with the higher  configured port priority is in the standby role at the time the command  is issued.

 

RP/0/0/CPU0:ios#mlacp reset priority Bundle-ether 1 
Sun Aug 28 16:12:58.110 BST

Warning: this will reset all negotiated mLACP sessions on Bundle-Ether1.

If traffic is flowing over this bundle, traffic loss WILL occur.

Proceed with priority reset? [confirm]

RP/0/0/CPU0:Aug 28 16:13:02.983 : BM-DISTRIB[1132]: %L2-BM-6-MLACP_BUNDLE_ACTIVE : This device is now the active device for Bundle-Ether1 

 

Brute Force

 

The brute force mechanism does not involve any priority changes. When using brute  force, the operational priority always matches the configured priority. A  switchover is performed as follows:

  • When  the bundle goes down on the active POA (e.g. the minimum-active links  threshold is breached), the LACP state machine is updated for any ports  that are not encountering problems on the active POA. (Any ports which  have lost connectivity to trigger the condition should already have been  deselected.)
  • These updates to the LACP states are transmitted over mLACP to the other POA.
  • A  final LACP packet is sent to the DHD on each of the links to the Active  POA to cause the DHD to deselect them as well (and select links to the  other POA in preference).
  • After  this, there is no further LACP transmission on those links. This means  that the state of these links is not being monitored by LACP.
  • To  recover from this situation, once the link that failed has recovered  (enough to be LACP Selected) LACP operations are restarted on the  remaining links.

 

Revertive Behavior

 

Revertive behavior means that the bundle effectively has a "primary" and a  "secondary" POA. The bundle is active on the primary POA whenever  possible. It is only active on the secondary POA when it is down on the  primary, or the secondary has more available links and the maximize threshold has been breached.

The calculation to determine which POA is the "primary" for the bundle is as follows.

  • If the POAs have different mlacp port-priority configuration for the bundle, the one with the higher priority configured (smaller number) is the primary.
  • If the configured bundle mLACP priorities match, the POA with the smaller mlacp node configuration in the redundancy group is the primary.
  • NB: Between two POAs in an RG sharing multiple bundles, it's possible for each POA to be a primary for some of the bundles.

Bu  default, when the primary POA recovers, it delays going active for 300  seconds to allow higher-layer ICCP-aware protocols (e.g. routing  protocols) to sync their state between devices. This helps avoid churn  and downtime at higher layers. The delay time can be modified using the  following configuration setting:

interface <bundle> mlacp switchover recovery-delay <time in seconds>

 

Non-revertive Behavior

 

Non-revertive behavior does not consider POAs to be "primary" and "secondary"; only  "active" or "standby". The configured priorities are only used to  determine which POA is initially active. After this point, the active POA remains active until it encounters a failure, or mlacp maximize settings are in effect and the peer router has more links or bandwidth available (see later for details).

So without mlacp maximize there is no "switchback" in this mode; when a POA recovers from a failure, it remains standby until the other POA fails.

This mode of operation causes least churn, and is the recommended option.

 

Notes

 

The configuration options map to the following:

  • Default: Dynamic priority management with non-revertive behavior.
  • brute-force: Brute force mechanism with revertive behavior.
  • revertive: Dynamic priority management with revertive behavior.

Two things to note about the available settings:

  • Brute  force operation is not compatible with non-revertive operation; it is  inherently revertive because the POA with the higher operational  priority is fixed by configuration. So a combination of these options is  not available.
  • If  the DHD is the higher priority LACP system the LACP port priorities set  by the POAs for the members are ignored. So the only mechanism that can  be used is brute force, regardless of the configuration. An alarm is  raised by the Bundle Infra on the bundle to indicate when this happens.

 

18. Switchover Triggers

 

The following events can trigger a switchover to the mLACP peer:

  1. Link failure: A port or link between the DHD and one of the POAs fails.
  2. Device failure: Meltdown or reload of one of the POAs, with total loss of connectivity (to the DHD, the core and the other POA).
  3. Core isolation:  A POA loses its connectivity to the core network and therefore is of no  value, being unable to forward traffic to or from the DHD.

NB: Other documents (e.g. from IOS) may discuss 5 failure cases, listed as A-E. These map onto the above as follows:

  1. Failure of upstream port on the DHD: Covered by link failure case.
  2. Failure of link between DHD and POA: Covered by link failure case.
  3. Failure of downstream port on the POA: Again considered a type of link failure in the above set.
  4. Node failure: Described here as device failure, to clarify that it describes the router failing.
  5. PE isolation: Named core isolation here.

The  following sections give more detail on each type of failure, how they  can be produced, and the expected results. Each of these events only  result in a switchover if the bundle is in (hot or cold) standby state  on the standby POA; otherwise the bundle will be Down on both routers.

 

 

This can be triggered for testing purposes by disabling an active bundle member when:

  • The bundle on standby POA is listed in an mLACP Standby state in show bundle.
  • There are no Standby links connected to the currently active POA.
  • Either of the following is the case:        
    • Once the link is removed, there is not enough bandwidth/links in the bundle on that POA to meet a configured minimum-active threshold.
    • Once the link is removed, the amount of links/bandwidth in the bundle is below the mlacp maximize threshold and the standby POA has more links/bandwidth available in the bundle than the active POA.

The member can be removed from the bundle using any of the following events:

  • Fiber pull at either end of the POA-DHD link. (NB: This is the most representative way of testing this scenario.)
  • Shut down the POA-DHD link on the DHD.
  • Shut down the POA-DHD link on the POA.
  • Remove the bundle membership configuration on either end of the POA-DHD link.

 

Device Failure

 

This  represents a meltdown of the router carrying the traffic; total loss of  the node. The only valid test trigger for this event is a hard reset  (power down) of the device, even a router reload is not truly  representative as the device will tend to go down in a series of  indeterminate stages.

 

Core Isolation

 

This  is an ICCP event meaning that connectivity to the network core has been  lost. The configuration for core connectivity monitoring is owned by  ICCP and not the Bundle Infra, so the following is only a brief  description of the basic steps required; please see ICCP documentation  for more details.

One or more "backbone interfaces" can be configured under the redundancy group for monitoring by ICCP:

RP/0/0/CPU0:ios(config)#redundancy iccp-group <id> 
RP/0/0/CPU0:ios(config-redundancy-iccp-group)# backbone interface <core-facing interface name>

A  core isolation event will be declared by ICCP when all the specified  interfaces are operationally down, or not present (e.g. node reload).  (If the interfaces do not exist or are down when they are configured,  core isolation is declared immediately. In all cases it can be cleared  by removing the backbone interface configuration.)

If  a POA experiences this event it sends dying gasp LACP packet to the DHD  on its member links and stops LACP negotiations on them, using the same  mechanism as if a brute force switchover was being performed. However,  it also changes port priorities on all links if dynamic priority  management is in effect (in accordance with the revertive or  non-revertive behavior described above) as appropriate.

 

19. User-Controlled Switchover

 

There  are a couple of configuration commands that can be used to bring a  bundle down on one POA, triggering a switchover to its mLACP peer.  Additionally, in some cases exec CLI commands can be used to control the  switchovers.

 

Bundle Interface Shutdown

 

The first is simply:

RP/0/0/CPU0:ios(config)# interface Bundle-Ether <x>
RP/0/0/CPU0:ios(config-if)# shutdown

This  brings down all the links in the bundle interface and forces a  switchover to the peer. This can be used in all cases to force a  switchover. However, this means that neither the LACP states nor even  the link states of the bundle members can be monitored while the bundle  is down.

This works for all switchover modes.

 

Bundle Interface Bundle-level Shutdown

 

If a dynamic priority management form of switchover is in use, i.e. the bundle is not configured to use  brute force switchover and the POAs are the higher priority LACP system,  there is an alternative command:

RP/0/0/CPU0:ios(config)# interface Bundle-Ether <x>
RP/0/0/CPU0:ios(config-if)# bundle shutdown

With  this slight variation, instead of bringing the member links down, they  are kept in LACP Standby state. The bundle is declared down so the mLACP  peer takes over. However, LACP continues to operate and the health of  the links can be verified before allowing them to come up again.

This  works for revertive and non-revertive switchover modes (not brute  force). NB: The DHD also has to be the lower priority LACP system,  otherwise brute force will implicitly be used.

 

Non-revertive Switchover CLI

 

CLI  commands are also available for performing a switchover on user demand.  To use these commands, the bundle must be using the default mLACP  switchover behavior (non-revertive). (Otherwise the bundle would have to  simply revert to the originally active router immediately.)

A switchover to the standby POA can be performed by issuing the following command on the active POA:

RP/0/0/CPU0:ios# mlacp switchover Bundle-Ether <x>

If required, the same operation can be performed on the standby POA to become active:

RP/0/0/CPU0:ios# mlacp switchback Bundle-Ether <x>

However, the switchback command could cause a bundle flap on the POA that is becoming active, so it is preferable to use the switchover command on the other POA if possible.

These commands can also be set with a delay, for the switchover to be performed at some point in the future. Some examples:

RP/0/0/CPU0:ios# mlacp switchover Bundle-Ether <x> in 5 minutes
RP/0/0/CPU0:ios# mlacp switchback Bundle-Ether <x> in 3 hours
RP/0/0/CPU0:ios# mlacp switchover Bundle-Ether <x> at 08:45:30

If it is necessary to cancel a scheduled switchover or switchback operation the following commands can be used. The switchover and switchback variants can be used per bundle, the scheduling variant affects all scheduled bundle actions.

 

RP/0/0/CPU0:ios# clear mlacp switchover Bundle-Ether <x>
RP/0/0/CPU0:ios# clear mlacp switchback Bundle-Ether <x>
RP/0/0/CPU0:ios# clear mlacp scheduling

 

mLACP Synchonization

 

Synchronization between mLACP peer devices occurs on a number of occassions.

  • When  connecting to a peer device mLACP will send a full (unsolicited) sync  of all its configuration and state for the relevant ICCP group. It will  also request a full sync from the peer device.
  • When  inconsistent or incompatible data relating to a foreign port or a  mutual object (i.e. a bundle or system setting) is received from a peer a  full resync is requested. The exception to this is the case where the  data is received as part of a resync in which case it is NAKd. See the  following section for more details.
  • When  incorrect data relating to a local port is received from a peer a full  resync of the local configuration and state is sent back to the peer.
  • When  the local node id is changed a full resync of local configuration and  state is sent to the peer device. This is done because the node id forms  part of the identifier for every port so it is necessary to purge the  old port data and replace it with the new.

The  XR implementation of mLACP always requests a full resync although the  protocol allows for resyncs to be requested for particular objects. This  is done for reliability reasons as well as practicalities of  implementation. Similarly when a resync is request is received, or when  sending an unsolicited sync, the XR implementation always responds with a  full resync of local configuration and state regardless of what was  requested. This is fully compatible with the behavior outlined in the  standard.

 

NAKing mLACP Messages

 

Messages can be NAKd for a number of reasons, including but not limited to

  • Invalid TLV contents (e.g. value out of range)
  • Incompatibility between local and peer device (e.g. clashing node id or different bundle ROID on each device)
  • Object or parent object referenced in the message is not found in the local database

As  explained above, typically when a message is received for which there  is some sort of issue a resync is first requested (or sent). If an issue  is detected within an incoming sync, or for problems such as an  incorrect ROID for a bundle which cannot be resolved with a resync, then  a message is NAKd.

When  a message is NAKd the object referred to the in the message is disabled  in some way. The behavior depends on the type of object.

  • If a NAK for a local port, bundle or system is received it is disabled for LACP operation
  • If a NAK for a remote port, bundle or system is sent it is removed from the local database
  • If a NAK for a remote bundle or system is sent the corresponding local object is disabled for LACP operation

All operations cascade to child objects.

To  re-enable an object after it has been NAKd there must be some change in  its configured state that causes a new Config TLV to be sent. This  resets the NAKd state, although it is possible that the new Config TLV  will be immediately NAKd again if the change is not acceptable either.

 

20. Configuration Changes

 

Certain  elements of mLACP configuration are key, and mismatches or even  configuration changes can have an impact on mLACP operation.

 

mLACP Node ID

 

The value configured for the mlacp node under the ICCP group submode is used in the LACP parameters for bundles  in that ICCP group. This value must be present for bundles in the ICCP  group to be operational, and must differ on each device in the ICCP  group.

When  this value is modified, the LACP session must be renegotiated on each  link of each bundle in the ICCP group. This causes the interface state  of all the bundles to flap.

 

mLACP System ID

 

The mlacp system priority and mlacp system mac values under the ICCP group are also required for bundles in the ICCP  group to be operational. These values must be the same on each router in  the ICCP group.

Once  again, these values are used in LACP negotiations, so changing them  causes all the bundles to flap. (This is why they must be the same on  all devices: Otherwise, when the Active device fails, all the LACP  sessions on the Standby have to be renegotiated using its system ID.)

 

21. Split Brain

 

If  the ICCP link between the POAs goes down but both POAs remain up, a  "split brain" situation is said to have occurred, meaning that the POAs  cannot exchange state information and are not aware of each other's  presence.

In  this scenario, both POAs would attempt to go active: From each one's  perspective, it appears the other has encountered a device-level  failure. This can be protected against in a limited manner in some  cases, using DHD control as described below.

 

22. DHD Control

 

In the recommended configuration,  the switchovers are controlled only by the POAs. The DHD is always  trying to make all links active, and the required set are kept in  standby by the POA.

However, if the DHD supports maximum-active links configuration, this can be used to protect against both POAs going  active in a split brain scenario if all of the following conditions are  met:

  • The DHD has the same number of links to each POA in the bundle.
  • The POAs are configured with minimum-active links <number of links to the DHD>.
  • The DHD is configured with maximum-active links <number of links to each POA>.

Additionally,  if either POA has brought down its links by brute-force (i.e. due to  brute-force behavior being in effect), forwarding on the bundle could  stop (i.e. the bundle could go down on both POAs) when the split brain  event occurs. Therefore dynamic priority management is recommended.

If  a split brain event occurs with this configuration present, the POAs  will continue to operate with the same priorities as they had before and  will both try to go active. However, the DHD will only allow the links  to the POA with higher link priorities to go active due to the  maximum-active links configuration.

NB:  A switchover cannot be coordinated between the POAs while split brain  is in effect. So if the active POA encounters a failure, there is no  guarantee that bundle will switch over to the other POA. (The exact  behavior depends on the number of bundle member links and the manner of  the failure.)

 

 

 

Syslog messages

 

The following lists all the possible syslog messages from the bundle infra specific to mLACP, along with their meanings. In addition many of the RED_MGMT messages relate to LACP and mLACP and may be relevant. Refer to the documentation for that message in the regular troubleshooting guide.

 

 

MLACP_CANNOT_SWITCHOVER: Could not perform mLACP switchover/switchback requested by user for bundle <name>: <reason>

 

A switchover request was triggered using the mlacp switchover <bundle> CLI, and the Bundle Infra has attempted to perform the switchover. However, the bundle was not in an appropriate state to switch over, so no action was taken. This could be because:

 

  • One of the mLACP peers is down.
  • The bundle is not active on the node being switched from.
  • The switchover behavior in operation is not non-revertive.

 

MLACP_CONNECT_FAILED: Failed to connect to another mLACP device in ICCP Group <id>. Reason: <reason>

 

The connection failed so mLACP will not operate over the specified ICCP group. Most likely this is be because no peer device is configured to run mLACP in this group. Other possibilities include

 

  • Version mismatch between the two devices. The output of show mlacp can be used to check the version the device is using.

  • There are more ICCP connections than can be supported currently set up; this can be checked in configuration and corrected if required. To retry the connection, remove and re-add the mlacp node configuration under the ICCP group.

 

MLACP_CONNECT: Connected to <LDP ID>

 

A connection to the specified mLACP peer device has been established.

 

MLACP_DISCONNECT: Disconnected from <LDP ID>

 

The connection to the peer device identified has been terminated.

 

MLACP_SYSTEM_ID_ARBITRATION: The system ID for ICCP group <id> has been established by arbitration

 

If the system ID was established by arbitration then there is a misconfiguration or unadvisable choice of configuration; a different mlacp system priority or mlacp system mac has been specified for the same ICCP group by different peers. One of the values is chosen for operational purposes, but if the peer who owns that value disconnects, the other device will stop using that value, which could trigger a bundle flap.

 

Correct the misconfiguration to clear the alarm by configuring the same value for those configuration items on each router in the RG.

 

MLACP_BUNDLE_MAC_ARBITRATION: The MAC address for <bundle name> has been selected through arbitration.

 

This is the same as the system ID arbitration alarm but for the mac-address configuration under the bundle interface. This should also be the same for a bundle on all devices operating that bundle.

 

MLACP_UNRESOLVABLE_MISCONFIG_DISCONNECT: Disconnecting from <LDP ID> due to an unresolvable misconfiguration: <reason>
MLACP_RESYNC_INCONSISTENCY_DISCONNECT: Disconnecting from <LDP ID due to an inconsistency in the mLACP data that could not be resolved with a resynchronization. Reason: <reason>

 

The configuration specified is mismatched between the two POAs.

 

To recover from this situation you must correct the configuration mismatch, and then remove and re-add an item of RG mLACP configuration (e.g. the mlacp node under the ICCP group submode) on both POAs.

 

MLACP_DEVICE_MISCONFIGURATION: <reason>

 

Correct the specified misconfiguration to clear this issue.

 

MLACP_ITEM_MISCONFIGURATION: <details>

 

This message indicates a possible issue, but can be safely ignored. If there is an ongoing issue, an MLACP_DEVICE_MISCONFIGURATION message will be emitted shortly afterwards.

 

MLACP_ROID_MISMATCH: The ROID (<value>) received from <LDP ID> for <bundle name>, does not match that expected (<value). Please ensure that the ROID for the bundle is the same on both devices.

 

In 4.0, the ROID of the bundle is generated in a set format from the bundle ID, in both the XR and IOS implementations. However, in future it is possible that the ROID will be configurable. So the current implementation emits this message if it gets an unexpected value for the ROID.

 

MLACP_CORE_ISOLATION: <bundle name> marked as isolated due to not being able to connect to the core.

 

ICCP has declared a core isolation event for the redundancy group. As a result, the bundle infra has declared this bundle as isolated, and will switch over to the standby POA if it is available.

 

MLACP_BUNDLE_PEERING: <bundle name> is peering with <LDP ID>
MLACP_BUNDLE_PEERING: <bundle name> is no longer peering with <LDP ID>

 

This message indicates that the peer device being used for operating mLACP on this bundle (or that there is no longer an mLACP peer for this bundle).

 

MLACP_BUNDLE_ACTIVE: This device is now the active device for <bundle>
MLACP_BUNDLE_ACTIVE: This device is no longer the active device for <bundle>

 

This message indicates that the local device has taken on the Active role for the bundle in question, or has surrendered that role to the peer. This may be an expected event or may indicate that some fault has occured to trigger a switchover. Investigate as appropriate.

 

 

 

Recovering from failures

 

If you've identified a problem and collected all the necessary information, or you've hit a known issue, there are some common steps you might be able to use to recover the testbed without needing to reload:

 

  • Shut/no shut the bundle or member that's causing a problem.
  • Unconfigure/reconfigure the bundle or member.
  • process restart bundlemgr_distrib

  • process restart mpls_ldp

  • Remove then add some mLACP configuration under the ICCP group on both POAs, e.g. the mlacp nodde (to cause a reconnection to the ICCP group by mLACP).

 

23.

Simple quick config blocks

 

The connections in this topology are as follows:

 

   DHD                      POA 1                     POA 2

Gi0/0/0/0 --------------- Gi0/0/0/0
Gi0/0/0/1 --------------- Gi0/0/0/1
Gi0/0/0/2
Gi0/0/0/3 ----------------------------------------- Gi0/0/0/0
Gi0/0/0/4 ----------------------------------------- Gi0/0/0/1
                          Gi0/0/0/2                 Gi0/0/0/2
                          Gi0/0/0/3 --------------- Gi0/0/0/3
                          Gi0/0/0/4 --------------- Gi0/0/0/4

 

 

POA 1
redundancy
 iccp
  group 1
   mlacp node 1
   mlacp system mac 000d.000e.000f
   mlacp system priority 1
   member
    neighbor 5.4.3.2
   !
  !
 !
!
interface Bundle-Ether1
 lacp switchover suppress-flaps 300
 mlacp iccp-group 1
 mac-address 0.deaf.0
 bundle wait-while 100
!
interface Loopback0
 ipv4 address 5.4.3.1 255.255.255.255
!
interface GigabitEthernet0/0/0/0
 description Connected to DHD Gi0/0/0/0
 bundle id 1 mode active
 lacp period short
 no shutdown
!
interface GigabitEthernet0/0/0/3
 description Connected to POA2 Gi0/0/0/3
 ipv4 address 1.2.3.1 255.255.255.0
 proxy-arp
 no shutdown
!
router static
 address-family ipv4 unicast
  5.4.3.2/32 1.2.3.2
 !
!
mpls ldp
 router-id 5.4.3.1
 discovery targeted-hello accept
 log
  neighbor
 !
 interface GigabitEthernet0/0/0/3
 !
!

 


 

 

POA 2
redundancy
 iccp
  group 1
   mlacp node 2
   mlacp system mac 000d.000e.000f
   mlacp system priority 1
   member
    neighbor 5.4.3.1
   !
  !
 !
!
interface Bundle-Ether1
 lacp switchover suppress-flaps 300
 mlacp iccp-group 1
 mac-address 0.deaf.0
 bundle wait-while 100
!
interface Loopback0
 ipv4 address 5.4.3.2 255.255.255.255
!
interface GigabitEthernet0/0/0/0
 description Connected to DHD Gi0/0/0/3
 bundle id 1 mode active
 lacp period short
 no shutdown
!
interface GigabitEthernet0/0/0/3
 description Connected to POA1 Gi0/0/0/3
 ipv4 address 1.2.3.2 255.255.255.0
 proxy-arp
 no shutdown
!
router static
 address-family ipv4 unicast
  5.4.3.1/32 1.2.3.1
 !
!
mpls ldp
 router-id 5.4.3.2
 discovery targeted-hello accept
 log
  neighbor
 !
 interface GigabitEthernet0/0/0/3
 !
!

 

 

 

On the DHD
interface Bundle-Ether1
 lacp switchover suppress-flaps 300
 bundle wait-while 100
!
interface GigabitEthernet0/0/0/0
 description Connected to POA1 Gi0/0/0/0
 bundle id 1 mode active
 lacp period short
 no shutdown
!
interface GigabitEthernet0/0/0/3
 description Connected to POA2 Gi0/0/0/0
 bundle id 1 mode active
 lacp period short
 no shutdown
!

 

 


 

 

 

 

 

 

Xander Thuijs, CCIE #6775

  • Principal Engineer ASR9000
Version history
Revision #:
1 of 1
Last update:
‎07-15-2013 05:05 AM
Updated by:
 
Labels (1)
Comments

Thanks a lot Xander for this outstanding document. I read that in XR 4.3.1 there is support to pseudo-MC-LAG (Active/Active). Would it be possible to elaborate on that point as well?

Thanks,

Michel.

Cisco Employee

Thanks for the feedback Michel!

yes let me get cranking on this, may take a week or so, but I will update the doc with that info.

it basically comes down to one POA is active for vlans 10-20 and the other one for 30-40 (something like that) vlan loadbalancing if you will.

regards

xander

New Member

Hi, Xander:

I'm implementing the VPLS with decoupled PWs for a customer (but with BGP/AD).

I have a bunch of ASR9ks where the VPLS PWs take place.

The DHDs are 6500s with a LACP etherchannel that seems to be working appropiately.

While the setup works correctly, the switchover recovery time is at around 30s (both ways).

I'm testing this by pinging from an SVI on one 6k to the other.

On the remote side of the PWs I see how the local side MAC address dissapears on switchover like it is supposed to, but it takes 30s to reappear.

As far as I can tell the aggregated ports everywhere are working just fine.

I have 2 questions: are the 30s of recovery time correct? In case you think they are not, do you have any suggestions?

Thanks in advance!

c.

Cisco Employee

Hi Carlos, smells like a STP convergence maybe? although MCLAG would not require that...

Can you find out where and why the traffic is getting dropped?

A rapid ping with timeout 0 gives a defined rate in the np counters and we cna potentially see if the issue is in the 9k or somewhere else/to continue finding the culprit.

These scenarios are probably easier handled by the TAC for the right follow up.

30 seconds seems long, I have seen better.

regards

xander

New Member

Xander:

You're right it's an STP issue. For some reason the 6500 doesn't see both channel links as part of the bundle, so when one goes down, the port-channel an SVIx go down for a bit, and looks like STP has to reconverge.

The weird  thing is that, though I was OK with the mLACP PoA side to see one of the etherchannel ports as 'standby', I was expecting the DHD side to have both etherchannel ports as up and part of the etherchannel.

I'll keep looking.

Thanks,

c.

New Member

Quick q: in a VPLS-mLACP scenario, should I be using different lacp port-priorities on the lacp etherchannel (PoA side) or the same?

Cisco Employee

I see, yeah that will be an STP issue if the bundle on the dhd goes down.

Here some tricks:

The bundle flapped on the POA

Bundle flaps are sometimes expected on mLACP events, but they are usually the result of misconfigurations. So the first thing to do is check that all your configuration is correct. Please refer to the configuration guidance for details, and specifically take note of the following:

  1. If a protocol flap is seen (e.g. OSPF) but not a flap in the state of the bundle interface itself, please refer to the feature-specific wiki and follow up with the team responsible for the feature.
  2. The mlacp system priority and mlacp system mac configured under redundancy iccp group <x> must be configured to be the same value on both devices in the RG.

  3. The same mac-address should  be configured on the bundle interface on both POAs.

  4. If the DHD supports bundle wait-while 100 (XR 4.0) or lacp fast-switchover (XR up to 3.9 & IOS) or equivalent, this should be configured on the DHD.

  5. If the DHD has bundle wait-while 100 configuration, lacp switchover suppress-flaps 300 should be configured on the POAs. Otherwise, lacp switchover suppress-flaps 2500 is required.

  6. The bundle should be configured with bundle wait-while 100 on the POAs.

  7. Check for any syslogs or bistate alarms that could indicate the cause of the flap.

The bundle flapped on the DHD

Bundle flaps on the DHD are usually expected during switchover events. If an XR 4.0 device is in use, you can configure bundle wait-while 100 and lacp switchover suppress-flaps 300 on the bundle to avoid a flap (assuming the POAs are set up correctly as above). If other DHD software provides similar functionality it can be used, otherwise a flap cannot be avoided.

regards

xander

Cisco Employee

Must be the same on the POA's Carlos.

cheers

xander

New Member

Noted. Thanks.

I read your comments on flaps this morning. Can't find a way to delay the port-channel flap on the 6500 (like you do on the ASR). At the moment, we're thinking portfast on the port-channel on 6500 (DHD side).

Cisco Employee

hey carlos,

if you dont have a loop (which you probably dont with mclag, then you can indeed set portfast!

if the dhd is not tunable, we have to live with this port flap which will cause the stp convergence.

I cant do anything about it from the 9k/xr side...

regards

xander

New Member

I think portfast will do it. Just have to wait for a MW to test it.

Will update here if everything goes well. Thanks for everything, Xander.

New Member

we have this particular setup running with a N7K vpc cluster as DHD. Each N7K has one link to the active POA and one link to the standby POA. Failover works like a charm.

New Member

N7k, should respond faster.

BTW, portfast did the trick, went down to about 1s failover time.

Cisco Employee

Thanks for this nice document.

Does the ASR9K (POA's)  needs to be directly connected, or the requirement is just to have IP connectivity between POA's.

Cisco Employee

Thank Sadath!

no need to be directly connected, as long as there is ip connectivity indeed.

regards!!

xander

New Member

Xander:

I'm curious about one thing regarding mLACP: when you configure it between 2 directly connected 9ks (haven't tried any other setup), the LDP adjacency goes from NSR:Y to NSR:N/A. Why is this? Is there functionality lost here?

RP/0/RSP0/CPU0:XXXXX#show mpls ldp neighbor  brief

Tue Feb 18 09:29:19.768 CST

Peer               GR  NSR  Up Time     Discovery  Address  IPv4 Label

-----------------  --  ---  ----------  ---------  -------  ----------

[snip]

X.X.X.X:0    Y   N/A  1w3d                2        6        3779

[snip]

RP/0/RSP0/CPU0:XXXXXX#

Thanks,

c.

Cisco Employee

this session for ICCP is a targeted LDP session, depending on your release, there may not be a full supprot for NSR/GR on LDP.

if the flag goes to N/A or N it just means that the iccp sessions is not protected against failover with NSR or GR.

same as with a routing protocol.

cheers!

xander

New Member

Hi, Xander:

I'm running 4.3.2 on this particular setup. Is this supported on 4.3.4 or 5.x?

Thank you,

c.

New Member

Hi Xander,

Can you explain in more detail how these 2 recovery timers work together:

LACP:                                      Operational

    Flap suppression timer:                  300 ms

<snip>

  mLACP:                                     Operational

<snip>

    Recovery delay:                          300 s

We are running 3750s as the DHDs connecting to ASR9ks, therefore we don't have access to any of the lacp switchover commands on the DHDs.  On the ASRs we have:

lacp switchover suppress-flaps 300

  mlacp switchover type revertive

mlacp switchover maximize links

bundle wait-while 100

In a situation where we had near constant flapping of the links, we found the switchovers were no longer working as expected and appeared to lose sync. What is a recommended timer config to avoid this issue if we don't have those commands on the DHDs?

Hi xander ,

 

I do have a question here regarding ICCP and MPLS LDP NSR.

MPLS LDP NSR stays operational when iccp group is present as below.

 

Peer LDP Identifier: 192.101.0.232:0
  TCP connection: 192.101.0.232:55427 - 192.101.0.230:646; MD5 on
  Graceful Restart: Yes (Reconnect Timeout: 120 sec, Recovery: 180 sec)
  Session Holdtime: 180 sec
  State: Oper; Msgs sent/rcvd: 3262/3250; Downstream-Unsolicited
  Up time: 01:47:31
  LDP Discovery Sources:
      TenGigE0/1/0/0
      Targeted Hello (192.101.0.230 -> 192.101.0.232, active/passive)
  Addresses bound to this peer:
      172.25.14.90   192.101.0.232  199.212.160.82  199.213.160.10  
  Peer holdtime: 180 sec; KA interval: 60 sec; Peer state: Estab
  NSR: DisabledClients: Session Protection, AToM, ICCP <<<<< ICCP present

 

When iccp group is removed, we see the below.

Peer LDP Identifier: 192.101.0.231:0
  TCP connection: 192.101.0.231:32200 - 192.101.0.230:646; MD5 on
  Graceful Restart: Yes (Reconnect Timeout: 120 sec, Recovery: 180 sec)
  Session Holdtime: 180 sec
  State: Oper; Msgs sent/rcvd: 92/93; Downstream-Unsolicited
  Up time: 00:00:33
  LDP Discovery Sources:
      TenGigE0/0/1/0
      Targeted Hello (192.101.0.230 -> 192.101.0.231, active/passive)
  Addresses bound to this peer:
      172.25.14.81   192.101.0.231  199.213.160.6  
  Peer holdtime: 180 sec; KA interval: 60 sec; Peer state: Estab
  NSR: Operational
  Clients: Session Protection, AToM

 

Now, does that mean ICCP & MPLS LDP NSR cannot work together?

 

Thanks

 

 

New Member

Hi Xander,

this post is really helpful, however i am also interesting in pseudo-MC-LAG option, as it is not very clear in the Configuration guide.

Thanks in advance.

Kind regards,

Nikola

Cisco Employee

Ah thanks for the idea!! yeah let us document that out also, always looking for new ideas as to what to cover next :)

give me a few and we'll get it out there!

cheers!

xander

New Member

Hi Xander,

Thanks alot for your post, but I appreciate if you can elaborate more about the supported L2VPN and L3VPN services topologies that can be deployed using MCLAG, I read in some Cisco documents that coupled mode is not supported in ASR9K, unlike what's posted here? and can you please provide me with the configuration required for these supported topologies?

Many Thanks

 

 

 

Cisco Employee

hi there,

yeah that is why I write my own documentation to make sure the right support and capabilities are reflected :) (not ideal situation i know, we're working on that).

I think you have a great question, and I think I have some detail on that. One of the things on my to do list is to make use-cases. I realize that most of the doc that I have out there just describe the technology and don't have actual use cases whereby you can take a design or topology with a config and details. I recognize the need, but it is a lot of work. Let me get on that.

For the time being what type of design are you looking at and I can let you know how to set that up, rahter then me providing you use cases today that you may not need or find useful.

My personal pref is not using MCLAG to be honest, I find the cluster technology with active active bundles way more appealing, but that is something personal, some folks swear by MCLAG :)

So any case let me know what your need is and I'll detail it out for you as much as possible.

cheers!

xander

 

New Member

 

 

 

 

 

I agree with Xander, although a cluster setup has its own quirks. The thing with MCLAG is that, since it is MPLS dependent, at times of network instability it can do some funky stuff, specially from DHD perpective. Cluster-based bundles seem to be much more stable. Just my opinion.

 

 

 

 

New Member

Hi xander,

Thanks alot for your reply and interest to help, I'm currently supporting a big ISP to figure out the best redundancy setup at the aggregation layer PE-CE, and MCLAG is one of the proposed solutions as the customer has many 7600 platforms deployed in the network besides of new ASR9Ks.

what we need is to provide a redundancy solution for P2P PW, VPLS, and L3VPN ccts terminated on the PE boxes, I found some presentations on Ciscolive365 site demonstrating different topologies besides some configuration examples in the configuration guides, but there are some points are still vague for me, and I think things will be be more clear if I can get a use case for the P2P PW, VPLS, and L3VPN cases with the configuratio needed in all cases.

Many Thanks

New Member

Hi xander,

 

do you have any update on the question below?

 

Regards

New Member

Hi, Xander:

 

Is an mLACP scenario supported (by TAC) when the POA links are on nv sats?

 

Thanks!

c.

Cisco Employee

hey carlos,

I dont see a technical reason why this would not work, but I am quite sure that this is a scenario that we haven't tested specifically, I'll verify.

I would give it a try and see how it works, if there is an issue, I would open a tac case to have this corrected in case so, because I think there is value to do this.

cheers!

xander

New Member

Hi,

Can you please confirm if coupled and decoupled refers to the usage of P2P group or bridge group respectively?

 

Best regards!

Cisco Employee

hi atif, the reference to coupled vs decoupled basically means that the state of the PW (up/down) is coupled (or not) to the MCLAG status of the bundle interface (active/standby).

p2p PW (vpws) or bridge domains (vpls) can use either coupled or decoupled mode(s).

cheers

xander
 

New Member

Hi,

In case of P2P xconnect, the default behavior is coupled mode; and i am not able to find the command to change the mode. Similarly, in case of bridge domain, the default behavior is decoupled mode; and i am not able to find the command to change the mode.

In IOS, the default behavior is always coupled mode; and i found an optional command to change the mode "status decoupled" under l2 vfi config mode.

Can you please provide configuration examples for coupled and decoupled mode in case of p2p xconnect and bridge-domain?

 

Regards!

New Member

Hi xander,

Update:

I found a command "coupled-mode" under bridge-domain in order to enable couple mode under bridge-domain which is by default disabled. However, as per my test results, it is not functioning in coupled mode. Moreover this command is not documented in command reference of ASR9K IOS XR 5.x.x. Can you please advise if it is supported? 

 

And after a lot of research, i couldn't find any command to disable the couple mode in xconnect p2p. Can you please confirm if it is supported?

 

Regards!

Cisco Employee

hi atif, yup it is supported since XR 4.2.1.

for Xcon it is not configurable, by default it is a coupled mode there.

Here is some extra detail :

 

VPLS coupled-mode implementation (signal down PW when access circuit is down)

Introduction

In a point-to-point cross connect (VPWS), the access circuit (AC) and the PW are coupled. So, if the AC goes down, the L2VPN PE signals via LDP to the remote PE that the PW status should be down. This triggers convergence when PW redundancy is configured.   The AC and PW are not coupled in a LAN service (VPLS) by default.  So if the AC is down, the PW will be up regardless of the status of the AC because it is in the decouple mode.

 

Prerequisites

In 4.2.1 the XR software team made the coupled-mode implementation for VPLS configurable (CSCtr24794).

 

Configure

l2vpn

 bridge group xxxx

  bridge-domain xxxx

   coupled-mode

   interface TenGigEx/x/x/x

   !

   interface TenGigEy/y/y/y

   !

   vfi vfi-XX

    neighbor x.x.x.x pw-id xxx

 

Verify

 

Couple-mode disabled (not Configured) - PW will be up

RP/0/RSP0/CPU0:9010-A-SVE1#show l2vpn bridge-domain bd-name accedian_data2 detail
Legend: pp = Partially Programmed.
Bridge group: accedian, bridge-domain: accedian_data2, id: 12, state: up, ShgId: 0, MSTi: 0
Coupled state: disabled

 

Coupled-mode enabled (AC circuits up) - PW will be up

RP/0/RSP0/CPU0:9010-A-SVE1#sh l2vpn bridge-domain bd-name accedian_data1 detail
Legend: pp = Partially Programmed.
Bridge group: accedian, bridge-domain: accedian_data1, id: 4, state: up, ShgId: 0, MSTi: 0
Coupled state: up

 

Coupled-mode enabled (AC circuits down) - PW will be signaled down

RP/0/RSP0/CPU0:9010-A-SVE1#sh l2vpn bridge-domain bd-name accedian_data1 detail
Legend: pp = Partially Programmed.
Bridge group: accedian, bridge-domain: accedian_data1, id: 4, state: up, ShgId: 0, MSTi: 0
Coupled state: down

 

Troubleshoot

This feature will not work with a VPWS setup with multiple interfaces in a bridge domain.  The work around is to setup a VPLS as a point-to-point circuit with only one PW neighbor endpoint.

New Member

Hi,

Thanks a lot. I will test it and will come back to you if any issue pops up.

 

regards!

New Member

Hi Xander,

 

I use MC-LAG with VPLS decoupled mode on ASR9k.

My DHD are Nexus with vPC.

 

I try to determine the theorical switchover time from the active POA to the standby POA.

Can you help me ?

Cisco Employee

L2 service: Convergence should be sub-second, for most of the asr9k related failures

Convergence also depends on the access DHD switch. For most
of the time, the slow convergence come from the access switch, not asr9k. For
example, most of the access switch LAG convergence depend on the VLAN scale

 

Testing result for the DCI deployment: N7K vPC + asr9k MC-LAG with 500 VLANs, it can get sub-second convergence. But with 1200 VLANs, it can't.

 

failure detection for failover, depends on the speed of the iccp signaling and detection time.

 

cheers

xander

New Member

Hi Xander,

 

I have less than 100 vlans and with the Nexus 5k in vPC, I don't have LAG convergence due to spanning-tree for example.

So I will consider that the switchover is sub-second.

 

Thank you so much.

 

Best regards,

Samuel

New Member

Hey Xander,

 

Great article on MC-LAG, thanks for all the detail! In testing Pseudo-MC-LAG (ICCP-SM) I've run into a few caveats:

  • If l2vpn iccp-sm config is added before the BE and bridge-domain are setup it remains in an 'unconfigured' state until at least one bridge-domain is built with an EFP from the BE attached and the iccp-sm config is removed and re-added. ICCP-SM seems very sensitive to the order the router is configured.

    Interface Name: Bundle-Ether3
      MAC flushing: MVRP
    Recovery Delay: 180 (Timer not running)
       Local State: Unconfigured
      Remote State: Unknown

     
  • Additionally when adding or changing primay/secondary vlans to the iccp-sm config the routers will go into a mis-configured state until both sides match. Is there any way to have the router wait a certain period before going into mis-configured. It appears that any vlan changes are service impacting at this point. Only work around for us at this point is to hit commit on both routers at the same time to limit outage to sub-second hit.

 

Any input you have is appreciated, thanks!

 

 

 

Cisco Employee

hey jon!!

ah the first one I know, pain in the you know what indeed! ddts for that is CSCuo63884, fixed in XR53. (doesnt seem to be externally visible via toolkit however).

the wait for time, hmm need to think about that how to resolve that, I dont have an answer for you on that right now.

cheers!

xander

New Member

Thanks for the input, good to know about the fix for the iccp-sm configuration order.

For the VLAN mis-configure problem, i'm sure this is expected behavior, kind of like a port-channel switchport trunk vlan mismatch. But if default behavior could just block the 'new' VLAN and keep forwarding the existing VLANs until the config is sync'd up that would allow us to make changes without impacting live traffic. This is probably a feature request, but sure would be nice if it were possible.

 

Regards,

Jon

 

New Member

Xander - great article!  We've used this config to connect many Calix devices via Layer 2 into a bridge-domain on each 9k (with VFIs connecting the BDs across each 9k and to the other side of the network).

We have now been tasked with coming up with a solution to terminate the Bundles at Layer 3 on each 9k instead of landing them into BDs.  Can we do the same Layer 3 IPv4 address on the bundle on each 9k?  This seems like it may not work because we need to also get that address into OSPF and may cause a duplicate IP in the routing table.  The other solution seems  to be to do HSRP between BVIs that are also attached to the BDs.

What are your thoughts on doing Layer 3 on the 9k?  Maybe we are missing something obvious...

-ben

Hello Xander,

Although some people here have mentioned interoperability between N7K vPC and ASR9K MC-LAG, I'd like to ask if we should take special care of something. vPC on one hand is active/active, MC-LAG on the other hand is active/standby.. Would you recommend such a setup or is it too complicated?

Considering the system priority, both vPC and MC-LAG guides recommend setting higher system priority than the peer. I guess in this case MC-LAG should have higher priority and DHD control should be configured on N7K, correct?

Thanks,

George

Cisco Employee

If the peer is in active/active and the asr9k in active/standby, you may see undesirable behaviour. The asr9k will never send a data frame down the standby link, but the LAG link state is not checked on received frames. We are evaluating a change in the code that will drop ingress packets on a standby link. We are threading carefully because of possible performance impact.

hth,

Aleksandar

Hi Aleksandar,

I had the impression that if MC-LAG cluster had higher priority than the vPC cluster, the vPC would also suspend the links connected to the standby POA.  If it doesn't, I don't see how MC-LAG could interoperate with vPC..

 

Cisco Employee

Hi Georgios,

your expectation is correct. If you set the asr9k system priority to be higher (lower value) than the N7k, the N7k will follow the link assignment set by the asr9k.
 

New Member

Great article! thanks.

To my understanding regarding ICCP Groups, it will always go down, if all the backbone interfaces goes down, and thereby also force the bundles to go down?

If so, that would mean that a ICCP Group and bundle will always go down on POA1, if POA2 power off, and they only have single direct link in between them. Because i looses all connectivity to network core and goes into isolation.


A solution to this would then be, to have additional backbone interfaces configured, which is not directly connected between POA1 and POA2. Then we would see a split-brain situation, but the ICCP group will stay up on POA1, because it still have an active backbone interface, when POA2 is off.

Can you verify that?

Thanks.

Christian

Cisco Employee

cheers christian! takes long time to write things like this so great to see it is worth the effort!! :)

if your ICCP session takes the path over the core interfaces then yes you are correct.

generally, there is a link of some sort directly betwene the POA's hence if the core ifs go down, it is not necessarily detected by ICCP.

If the POA's can't see each other anymore, it is somewhat tricky, because it might lead to split brain (aka both POA's assume the other is not there and both being active).

it is best to have a "dedicated" channel between both POA's, and use some event detection to identify the core going down and trigger a POA failover with a priority change or some forced switchover in that facinity. EEM can help here or ojbect tracking can also facilitate here.

cheers!

xander

New Member

Hi Xander,

i have a question, here is our current topology and already operational.

We implement MC-LAG from our P1 and P2 connect to PE02. Named Bundle-Ether 25, contain l2vpn traffic on it.

P1 (BE25)------PE02 (BE25)

P2 (BE25)------PE02 (BE25)

My question, is that possible if we create a new MC-LAG on the same chassis to the same destination? The future topology should be look like :

P1 (BE25)------PE02 (BE25) for l2vpn traffic

P2 (BE25)------PE02 (BE25) for l2vpn traffic

P1 (BE30)------PE02 (BE30)

P2 (BE30)------PE02 (BE30)

Thanks.

Cisco Employee

ICCP peering is established between LDP router-IDs. Since you can't have multiple LDP router-IDs on a single router, the configuration that you are considering is not supported. However, if you deploy pseudo-MC-LAG, you would have an active/active scenario in which you can deploy per-VLAN load balancing.

hope this helps,

Aleksandar