Etherchannel and STP

Answered Question
Sep 27th, 2010
User Badges:

Recently there was a debate among some folks about how STP is handled within an Etherchannel. I understand that once an Etherchannel is defined, regardless of PAGP or LACP, that the new port-channel is introduced into the spanning tree and the physical ports are removed. That makes sense. The question is, how are BPDUs passed across the port-channel? I found an article (below) that references that for 2900s and 3550s, it uses the default port. That makes sense. But I can't find any documentation for other switches. 3560s? 4k? 6k? Nexus? Is the same functionality applied across the product lines?


http://www.cisco.com/en/US/customer/tech/tk389/tk213/technologies_tech_note09186a0080094714.shtml


2900/2950/2550 section:


The default port is used to transmit       traffic, such as Spanning Tree Protocol (STP), multicasts, and       unknown unicasts. The default port can be identified from the       output of the command show etherchannel summary by a       notation of d.

Correct Answer by krahmani323 about 6 years 9 months ago

Hello community,


Thanks for the intersting information.


Regarding the very last point about Etherchannel and STP cost, I have seen in this link => http://www.cisco.com/en/US/products/hw/switches/ps700/products_white_paper09186a00801b49a4.shtml#cg6  (check Spanning Tree Port Cost Calculation part) that :


- In CatOS, the spanning-tree port cost for an EtherChannel stays the           same after the port channel member link failure.

- In Cisco IOS Software, the           port cost for the EtherChannel is updated immediately in order to reflect the           new available bandwidth.




Interface

STP Port Cost

Short Mode (16-Bit)

STP Port Cost

Long Mode (32-Bit)

100-Mbps Ethernet

19

200,000

GE

4

20,000

Two-port Gigabit EtherChannel (GEC)

3

10,000

Three-port GEC

2

6666

Four-port GEC

2

5000

Five-port GEC

2

4000

Six-port GEC

2

3333

Seven-port GEC

2

2857

Eight-port GEC

1

2500

10-GE

2

2000

Two-port 10-Gigabit EtherChannel

1

1000




This discussion was also intersting about Eterchannel STP cost calculation => https://supportforums.cisco.com/message/3107809#3107809

===================================================

An internal formula is used to determine the default cost for an etherchannel which leads to the following default values:


If  you want more granularity and still have the dynamic cost update, you  can change the coding from 16bits to 32bits. In this case you will have a  unique value based on the number of links and the speed of the link.  You need to configure spanning-tree pathcost method long on the swith.

===================================================



Regards.

Karim

Correct Answer by Peter Paluch about 6 years 9 months ago

Dan,


The STP "cost" of a portchannel doesn't change when a member link goes 
down


I believe that this is an incorrect statement. The STP cost of a Port-channel interface does change if a member link goes down. In fact, the first thing to look at is the current Port-channel bandwidth value. If N links of capacity C are bundled in a single EtherChannel, Cisco Catalyst switches assign the default bandwidth of N x C to the corresponding Port-channel interface. The STP cost of this interface is subsequently computed from this value. If a member link goes down, the bandwidth of the Port-channel will be automatically updated and thus the STP cost will change as well.


I have verified and confirmed this behavior on 2950, 2960, 3560 and 3560V2 in our lab just now, and I am enclosing a small commented transcript of what does it look like:


Sw2#show cdp n
Capability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge
                  S - Switch, H - Host, I - IGMP, r - Repeater, P - Phone

Device ID        Local Intrfce     Holdtme    Capability  Platform  Port ID
Sw1              Fas 0/23           160          S I      WS-C2950T-Fas 0/23
Sw1              Fas 0/24           160          S I      WS-C2950T-Fas 0/24

Sw2#show ether sum
Flags:  D - down        P - in port-channel
        I - stand-alone s - suspended
        H - Hot-standby (LACP only)
        R - Layer3      S - Layer2
        u - unsuitable for bundling
        U - in use      f - failed to allocate aggregator
        d - default port

Number of channel-groups in use: 1
Number of aggregators:           1

Group  Port-channel  Protocol    Ports
------+-------------+-----------+-----------------------------------------------
1      Po1(SU)         LACP      Fa0/23(Pd)  Fa0/24(P)

Sw2#show run int fa0/23
Building configuration...

Current configuration : 63 bytes
!
interface FastEthernet0/23
channel-group 1 mode active
end

Sw2#show run int fa0/24
Building configuration...

Current configuration : 63 bytes
!
interface FastEthernet0/24
channel-group 1 mode active
end

Sw2#show int po1 | i BW
  MTU 1500 bytes, BW 200000 Kbit, DLY 1000 usec,
Sw2#show span int po1

Vlan             Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
VLAN0001         Root FWD 12        128.65   P2p
Sw2#
00:05:05: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/23, changed state to down
00:05:06: %LINK-3-UPDOWN: Interface FastEthernet0/23, changed state to down
Sw2#show int po1 | i BW
  MTU 1500 bytes, BW 100000 Kbit, DLY 1000 usec,
Sw2#show span int po1

Vlan             Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
VLAN0001         Root FWD 19        128.65   P2p
Sw2#
00:05:49: %LINK-3-UPDOWN: Interface FastEthernet0/23, changed state to up
00:05:53: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/23, changed state to up
Sw2#show int po1 | i BW
  MTU 1500 bytes, BW 200000 Kbit, DLY 1000 usec,
Sw2#show span int po1

Vlan             Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
VLAN0001         Root FWD 12        128.65   P2p
Sw2#


Note that originally, the bandwidth of the interface Po1 was set to 200000 Kbit/s (200 Mbps - two FastEthernet ports bundled together), and the resulting STP cost was 12 (running PVST). After I disconnected Fa0/23, the bandwidth decreased automatically to 100000 Kbps and the STP cost was updated to 19 - the common STP cost for a single FastEthernet link. When I returned the link back, the bandwidth was increased back to 200 Mbps and the STP cost was again updated to 12.


Surely, this is not a mechanism of STP alone, rather, this is a cooperation of various mechanisms in the IOS - one of them constantly watches the number of active links in an EtherChannel bundle and updates the default bandwidth on the Port-channel interface, the second - STP - merely reacts to the interface bandwidth modifications.


Best regards,

Peter

Correct Answer by Giuseppe Larosa about 6 years 9 months ago

Hello Chenson,


what I would say is: STP can update the member link cost as soon as some mechanism detects that one member link is down, but STP does not provide member link detection as the links are seen as a whole:

if a single BPDU is sent in each direction and always uses the same link, only those two links are "monitored" by STP not the other members of the bundle.


Hope to help

Giuseppe

  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 5 (4 ratings)
Loading.
Giuseppe Larosa Mon, 09/27/2010 - 11:22
User Badges:
  • Super Silver, 17500 points or more
  • Hall of Fame,

    Founding Member

Hello CHenson,


implementation of ethechannel on C2950 is peculiar and I would not consider it for higher end switches.


STP traffic can be subject to L2 hashing algorithm as they are L2 frames with a multicast destination and so it can use a different member link in different directions. ( on higher end switches I have this impression)


What is important to note is that STP does not handle ALL member links at the same time, and so the use of LACP ( I would consider PAGP legacy now) is highly recommended combined with STP.


Hope to help

Giuseppe

chenson Mon, 09/27/2010 - 11:39
User Badges:

Hello Giuseppe. Thanks for the reply.


I understand your position on the 2950 so that makes sense. However, not being able to find anything else on the other platforms confused me.


As for the STP hashing, you are saying that because a BPDU has a MAC in the frame (and the destination is mcast), it is subject to the same traffic distribution as other packets in the port-channel? OK. That makes sense as well.


Your last statement confuses me. "STP does not handle ALL member links at the same time". Can you elaborate? STP does not see the member links right? Regardless of PAGP or LACP?


NOTE: Yes, we are an LACP standard shop. PAGP is not encouraged in our environment.


--chenson

Correct Answer
Giuseppe Larosa Mon, 09/27/2010 - 12:19
User Badges:
  • Super Silver, 17500 points or more
  • Hall of Fame,

    Founding Member

Hello Chenson,


what I would say is: STP can update the member link cost as soon as some mechanism detects that one member link is down, but STP does not provide member link detection as the links are seen as a whole:

if a single BPDU is sent in each direction and always uses the same link, only those two links are "monitored" by STP not the other members of the bundle.


Hope to help

Giuseppe

danrya Tue, 09/28/2010 - 17:37
User Badges:
  • Bronze, 100 points or more

The STP "cost" of a portchannel doesn't change when a member link goes down.  STP is only monitoring the reachability through the port channel, not individual links.  When a member link "goes down" the link will be removed from the port channel and STP will NOT need to re-converge, since the "port" is still up and active. An STP event will not occur unless all member ports go "down".


I would recommend PAGP or LACP for the port channel.


You are correct that the BPDU's are hashed just like any other traffic, and will be assigned to a particular member link, but if that member link is removed, a new hash is performed, and it's assigned to another link.


Dan

Correct Answer
Peter Paluch Wed, 09/29/2010 - 01:20
User Badges:
  • Cisco Employee,

Dan,


The STP "cost" of a portchannel doesn't change when a member link goes 
down


I believe that this is an incorrect statement. The STP cost of a Port-channel interface does change if a member link goes down. In fact, the first thing to look at is the current Port-channel bandwidth value. If N links of capacity C are bundled in a single EtherChannel, Cisco Catalyst switches assign the default bandwidth of N x C to the corresponding Port-channel interface. The STP cost of this interface is subsequently computed from this value. If a member link goes down, the bandwidth of the Port-channel will be automatically updated and thus the STP cost will change as well.


I have verified and confirmed this behavior on 2950, 2960, 3560 and 3560V2 in our lab just now, and I am enclosing a small commented transcript of what does it look like:


Sw2#show cdp n
Capability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge
                  S - Switch, H - Host, I - IGMP, r - Repeater, P - Phone

Device ID        Local Intrfce     Holdtme    Capability  Platform  Port ID
Sw1              Fas 0/23           160          S I      WS-C2950T-Fas 0/23
Sw1              Fas 0/24           160          S I      WS-C2950T-Fas 0/24

Sw2#show ether sum
Flags:  D - down        P - in port-channel
        I - stand-alone s - suspended
        H - Hot-standby (LACP only)
        R - Layer3      S - Layer2
        u - unsuitable for bundling
        U - in use      f - failed to allocate aggregator
        d - default port

Number of channel-groups in use: 1
Number of aggregators:           1

Group  Port-channel  Protocol    Ports
------+-------------+-----------+-----------------------------------------------
1      Po1(SU)         LACP      Fa0/23(Pd)  Fa0/24(P)

Sw2#show run int fa0/23
Building configuration...

Current configuration : 63 bytes
!
interface FastEthernet0/23
channel-group 1 mode active
end

Sw2#show run int fa0/24
Building configuration...

Current configuration : 63 bytes
!
interface FastEthernet0/24
channel-group 1 mode active
end

Sw2#show int po1 | i BW
  MTU 1500 bytes, BW 200000 Kbit, DLY 1000 usec,
Sw2#show span int po1

Vlan             Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
VLAN0001         Root FWD 12        128.65   P2p
Sw2#
00:05:05: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/23, changed state to down
00:05:06: %LINK-3-UPDOWN: Interface FastEthernet0/23, changed state to down
Sw2#show int po1 | i BW
  MTU 1500 bytes, BW 100000 Kbit, DLY 1000 usec,
Sw2#show span int po1

Vlan             Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
VLAN0001         Root FWD 19        128.65   P2p
Sw2#
00:05:49: %LINK-3-UPDOWN: Interface FastEthernet0/23, changed state to up
00:05:53: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/23, changed state to up
Sw2#show int po1 | i BW
  MTU 1500 bytes, BW 200000 Kbit, DLY 1000 usec,
Sw2#show span int po1

Vlan             Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
VLAN0001         Root FWD 12        128.65   P2p
Sw2#


Note that originally, the bandwidth of the interface Po1 was set to 200000 Kbit/s (200 Mbps - two FastEthernet ports bundled together), and the resulting STP cost was 12 (running PVST). After I disconnected Fa0/23, the bandwidth decreased automatically to 100000 Kbps and the STP cost was updated to 19 - the common STP cost for a single FastEthernet link. When I returned the link back, the bandwidth was increased back to 200 Mbps and the STP cost was again updated to 12.


Surely, this is not a mechanism of STP alone, rather, this is a cooperation of various mechanisms in the IOS - one of them constantly watches the number of active links in an EtherChannel bundle and updates the default bandwidth on the Port-channel interface, the second - STP - merely reacts to the interface bandwidth modifications.


Best regards,

Peter

Correct Answer
krahmani323 Wed, 09/29/2010 - 02:54
User Badges:
  • Silver, 250 points or more

Hello community,


Thanks for the intersting information.


Regarding the very last point about Etherchannel and STP cost, I have seen in this link => http://www.cisco.com/en/US/products/hw/switches/ps700/products_white_paper09186a00801b49a4.shtml#cg6  (check Spanning Tree Port Cost Calculation part) that :


- In CatOS, the spanning-tree port cost for an EtherChannel stays the           same after the port channel member link failure.

- In Cisco IOS Software, the           port cost for the EtherChannel is updated immediately in order to reflect the           new available bandwidth.




Interface

STP Port Cost

Short Mode (16-Bit)

STP Port Cost

Long Mode (32-Bit)

100-Mbps Ethernet

19

200,000

GE

4

20,000

Two-port Gigabit EtherChannel (GEC)

3

10,000

Three-port GEC

2

6666

Four-port GEC

2

5000

Five-port GEC

2

4000

Six-port GEC

2

3333

Seven-port GEC

2

2857

Eight-port GEC

1

2500

10-GE

2

2000

Two-port 10-Gigabit EtherChannel

1

1000




This discussion was also intersting about Eterchannel STP cost calculation => https://supportforums.cisco.com/message/3107809#3107809

===================================================

An internal formula is used to determine the default cost for an etherchannel which leads to the following default values:


If  you want more granularity and still have the dynamic cost update, you  can change the coding from 16bits to 32bits. In this case you will have a  unique value based on the number of links and the speed of the link.  You need to configure spanning-tree pathcost method long on the swith.

===================================================



Regards.

Karim

Giuseppe Larosa Wed, 09/29/2010 - 12:46
User Badges:
  • Super Silver, 17500 points or more
  • Hall of Fame,

    Founding Member

Hello Karim,

you did a very good job in this post clarifying some points that had caused some useful discussion about STP capability or not to update cost of the bundle.


Keep go on in this way


Best Regards

Giuseppe

chenson Wed, 09/29/2010 - 12:54
User Badges:

Agreed. Thank you all for your comments and input. It really helped me understand this.

krahmani323 Thu, 09/30/2010 - 07:18
User Badges:
  • Silver, 250 points or more

Hi,


Thank you for your message and feedbacks, I appreciate.

And thanks for the nice rating.


Regards.
Karim

David Williams Thu, 09/30/2010 - 06:20
User Badges:

For what it is worth, the Nexus 5010 does not appear to update the  spanning-tree cost when a Port channel member link drops.  The IOS  upstream device does but not the Nexus.  I'm hoping to find a command to  change that behavior but I'm not very optimistic.  I haven't had much luck finding anything that might change that behavior.


So  judging by this post and my testing CATOS does NOT adjust spanning tree  cost with the loss of a member link. IOS DOES adjust spanning tree cost  with the loss of a member link. and NX-OS goes backwards and does NOT  adjust spanning tree cost with the loss of a member link. (At least not  by default)  I'm assuming they figured you would be using vss or vpc and  thus didn't commit the dev resources.  Sure would have been nice.  If  anyone can contradict me please do so!  I really want to be wrong about this. 

chenson Thu, 09/30/2010 - 06:40
User Badges:

Sadly, the more I think about it the more it makes sense. Meaning, I can see both viewpoints.


Viewpoint A - "If I lose a leg from my channel I want STP ro appropriately adjust. These things happen."


Viewpoint B - "If I lose a leg from my channel there is a problem that I need to correct. In the mean time, DON'T adjust STP as I don't want it to flip anything around unintentionally."


I wonder if Veiwpoint B is the Cisco school of thought.....

danrya Thu, 09/30/2010 - 06:56
User Badges:
  • Bronze, 100 points or more

I was about to type the same thing.


From a "stability" stand point, I wouldn't want an STP convergence on my core due to a single link failure on a port channel.  By updating the cost of the port channel I could pick a different path.  If it's the port channel to the Root I don't think I would want that to happen, since all switches would have to recalculate the tree.


On IOS if you want stability you can manually set the STP cost on the etherchannel, and then it will not cause a spanning tree convergence when a link is lost.  So, in IOS you can have it either way.

chenson Thu, 09/30/2010 - 07:08
User Badges:

Agreed. But a little consistency from Cisco wouldn't hurt either.

David Williams Thu, 09/30/2010 - 07:39
User Badges:

Both are good points. I could see how you could approach it from either direction.  In our case we would prefer the  convergence over the choke point given the preferred sizing of the port  channel. Perhaps the correct approach is to oversize the port channel to  the point that the loss of an upstream switching card or a member link  or two doesn't create a choke point.  I hate to over commit ports so much but maybe that is just the cost of redundancy.


I appreciate the feedback.

chenson Thu, 09/30/2010 - 07:45
User Badges:

It's a loaded question.


Do you use truly use LACP for thruput gains or link redundancy? In our case, my focus in the design is link redundancy. I always scale to N+1. So in the event of a cable failure (rare but it does happen) I don't sacrifice thruput. No choke point. I admit, it costs me on port capacity but in most environments that is an acceptable hit for the availability guarantee.


On the same note, one of our bigger debates going on internally is whether to move to 10G cards in some of our closets where we are up to 4 or 6 cables in a bundle. Cost vs Performance vs etc....

Actions

This Discussion

Related Content