show interface status | include Po

STUART KENDRICK · ‎02-08-2010

What is the CLI trying to tell me when it talks about EtherChannels with an 'A' or 'B' designator following them? i.e. I understand 'Po41' ... but what are these 'Po41A' and 'Po42B' thingies? It seems to be a way of representing multiple ports in a bundle ... but I don't see why it is useful. Nor why the alpha designator is sometimes 'A' and sometimes 'B'. Is this conveying information which I want to know?

cat6513#show interface status | include Po
Port           Name                     Status         Vlan      Duplex     Speed
Po40         To tungsten-a e1b    connected    42         a-full        10G
Po41         To tungsten-a e0b    connected    42         a-full         a-1000
Po41A                                     connected    42         a-full         a-1000
Po42         To tungsten-b e0a    connected    42         a-full         a-1000
Po42B                                     connected    42         a-full         a-1000
Po43         To tungsten-b e4c    connected    42         a-full         a-1000

--sk

cat6513# show etherchannel summary

Group Port-channel Protocol Ports

[...]

40       Po40(SU)        LACP      Te5/4(P)
41       Po41(SU)        LACP      Gi7/42(P)
41       Po41A(SU)      LACP      Gi7/43(P)      Gi7/44(P)
42       Po42(SU)        LACP      Gi7/40(P)
42       Po42B(SU)      LACP      Gi7/41(P)
43       Po43(SU)        LACP      Gi7/39(P)
[...]

cat6513# show lacp internal

[...]

Channel group 40
                            LACP port     Admin     Oper    Port        Port
Port      Flags   State     Priority      Key       Key     Number      State
Te5/4     SA      bndl      32768         0x28      0x28    0x505       0x3D

Channel group 41
                            LACP port     Admin     Oper    Port        Port
Port      Flags   State     Priority      Key       Key     Number      State
Gi7/42    SA      bndl      32768         0x29      0x29    0x72B       0x3D
Gi7/43    SA      bndl      32768         0x29      0x1029 0x72C       0x3D
Gi7/44    SA      bndl      32768         0x29      0x1029 0x72D       0x3D

Channel group 42
                            LACP port     Admin     Oper    Port        Port
Port      Flags   State     Priority      Key       Key     Number      State
Gi7/40    SA      bndl      32768         0x2A      0x2A    0x729       0x3D
Gi7/41    SA      bndl      32768         0x2A      0x102A 0x72A       0x3D

Channel group 43
                            LACP port     Admin     Oper    Port        Port
Port      Flags   State     Priority      Key       Key     Number      State
Gi7/39    SA      bndl      32768         0x2B      0x2B    0x728       0x3D

[...]

interface Port-channel40
description To tungsten-a e1b
switchport
switchport access vlan 42
switchport mode access
logging event bundle-status
spanning-tree portfast edge trunk
spanning-tree guard none
!
interface Port-channel41
description To tungsten-a e0b e4c e4d
switchport
switchport access vlan 42
switchport mode access
logging event bundle-status
spanning-tree portfast edge trunk
spanning-tree guard none
!
interface Port-channel42
description To tungsten-b e0a e4a
switchport
switchport access vlan 42
switchport mode access
logging event bundle-status
spanning-tree portfast edge trunk
spanning-tree guard none
!
interface Port-channel43
description To tungsten-b e4c
switchport
switchport access vlan 42
switchport mode access
logging event bundle-status
spanning-tree portfast edge trunk
spanning-tree guard none

!

[...]

interface TenGigabitEthernet5/4
description To tungsten-a e1b
switchport
switchport access vlan 42
switchport mode access
spanning-tree portfast edge
spanning-tree guard none
channel-protocol lacp
channel-group 40 mode active
!

[...]

interface GigabitEthernet7/39
description To tungsten-b e4c
switchport
switchport access vlan 42
switchport mode access
spanning-tree portfast edge
spanning-tree guard none
channel-protocol lacp
channel-group 43 mode active
!
interface GigabitEthernet7/40
description To tungsten-b e0a
switchport
switchport access vlan 42
switchport mode access
spanning-tree portfast edge
spanning-tree guard none
channel-protocol lacp
channel-group 42 mode active
!
interface GigabitEthernet7/41
description To tungsten-b e4a

switchport
switchport access vlan 42
switchport mode access
spanning-tree portfast edge
spanning-tree guard none
channel-protocol lacp
channel-group 42 mode active
!
interface GigabitEthernet7/42
description To tungsten-a e0b
switchport
switchport access vlan 42
switchport mode access
spanning-tree portfast edge
spanning-tree guard none
channel-protocol lacp
channel-group 41 mode active
!
interface GigabitEthernet7/43
description To tungsten-a e4c
switchport
switchport access vlan 42
switchport mode access
spanning-tree portfast edge
spanning-tree guard none
channel-protocol lacp
channel-group 41 mode active
!
interface GigabitEthernet7/44
description To tungsten-a e4d
switchport
switchport access vlan 42
switchport mode access
spanning-tree portfast edge
spanning-tree guard none
channel-protocol lacp
channel-group 41 mode active

Kevin Dorrell · ‎02-17-2010

I didn't know about this, but here is what I found out:

Aparently this means that within your bundles you have ports that are mutually incompatibly configured. It creates a series of bundles each grouping ports tghat are multually compatible. These are known as the secondary aggregators. It then aggegates the bundles into a super-bundle, known as the primary aggregator.

This reference looks pretty useful:

http://www.cisco.com/en/US/tech/tk389/tk213/technologies_configuration_example09186a0080094470.shtml#po1a

Also, from the command reference:

For LACP, multiple aggregators are supported. For example, if two different bundles are created, Po1 indicates the primary aggregator, and Po1A and Po1B indicates the secondary aggregators.

http://www.cisco.com/en/US/docs/switches/lan/catalyst6500/ios/12.1E/native/command/reference/show1.html#wp2017224

At first glance I don't see how your port configs are incompatible, but there might be some restrictions on your card, regarding which ports you can bundle.

Hope this helps

Kevin Dorrell

Luxembourg

STUART KENDRICK · ‎04-01-2010

OK, thanx for the links. The 'A' and 'B" designators indicate 'secondary aggregators', which IOS creates as part of an automagic effort to keep the LACP channel functioning, despite incompatible interface parameters.

Peering at my logs:

Mar 22 11:03:03 j4sr-b-esx 23230: 023225: Mar 22 11:03:02.645 pdt:
%EC-SP-5-CANNOT_BUNDLE_LACP: Gi7/42 is not compatible with aggregators
in channel 41 and cannot attach to them (flow control send of Gi7/42 is off, Gi7/44 is on)

And in fact, we can see this in 'sh int' output:

switch-a#sh int flowcontrol module 7
Port Send FlowControl Receive FlowControl RxPause TxPause
admin oper admin oper

----------- -------- -------- -------- -------- ------- -------
[...]
Gi7/42 desired off off off 0 0

Gi7/43 desired on off off 0 0

Gi7/44 desired on off off 0 0

[...]

Looking at the host side of things (NetApp filer/ONTAP), I can see that ONTAP is configured to auto-negotiate *and* has autonegotiated to 'flowcontrol full', which means 'send and receive' (http://ecserv1.uwaterloo.ca/netapp/man/man1/na_ifconfig.1.html) While the IOS is configured to autonegotiate (but asymmetrically, i.e. to prefer to send PAUSE frames but not to receive them).

toaster-a> ifconfig -a
[...]
e0b: flags=948043 mtu 1500
        ether 02:a0:98:12:db:1e (auto-1000t-fd-up) flowcontrol full
        trunked dmmvif3
[...]
e4c: flags=948043 mtu 1500
        ether 02:a0:98:12:db:1e (auto-1000t-fd-up) flowcontrol full
        trunked dmmvif3
e4d: flags=948043 mtu 1500
        ether 02:a0:98:12:db:1e (auto-1000t-fd-up) flowcontrol full
        trunked dmmvif3
[...]

I'm guessing that auto-negotiation isn't behaving the way I want it to, namely, I want ONTAP to honor the switch's request not to receive PAUSE frames.

Nevertheless. I don't think this matters -- i.e. what does the switch care if the host sends PAUSE frames? It isn't listening.

The part which kicks off the LACP oddness relates to how Gi7/42 sees the host wanting to *disable* its receipt of PAUSE frames ... and then of course the switch dutifully turns 'off' its sending of PAUSE frames, Gi7/43 and Gi7/44 stay as they were, and now the LACP channel contains three interfaces which disagree on flowcontrol settings.

I think the question becomes: is IOS disabling the transmission of PAUSE frames on a whim? Or is the host actually encoding its distaste for receiving PAUSE frames in the FLPs it is sending?

I don't even know what kind of tool I would need to analyze this ... I have in-line sniffers (Finisar THG and Cace TurboCap) ... but these boxes don't see FLPs.

Suggestions on how I could dig deeper into this?

--sk

STUART KENDRICK · ‎05-10-2010

OK, so I've chewed a little deeper into this. The folks who write the driver for the NIC in the host believe they have a flow-control bug, which results in the mismatch between flow control settings. LACP wants all members to be configured identically, so it doesn't want to bundle the 'odd' link. Thus far, I think my story is fairly solid. Where I stumble is on what happens next -- looks like the IOS, rather than yanking these members, creates these 'secondary' aggregates, which it then bundles into a primary aggregate ... effectively hiding the problem from us. But here I'm speculating.

--sk