Cisco Support Community
cancel
Showing results for 
Search instead for 
Did you mean: 
Announcements

Welcome to Cisco Support Community. We would love to have your feedback.

For an introduction to the new site, click here. And see here for current known issues.

New Member

6120 Port-Channel

Hello -

We migrated from a 6120HA pair with a single connection between 6120A and a nexus 5020 and a single connection between 6120B and our other nexus 5020 to a port-channel between each of our FIs and 5020s (staight LACP, no vpc on nexus side).  The nexus switches are connected to each other with a 2x10gb Vpc peer link.  They are connected with a vpc to the 6509 VSS system at our core.

Each UCS chassis is doubly connected to its respective side of the FI.  FI A is connected strictly to the left side FEXs and FI B to the right side FEXs.

This morning we had a connection flap up and down a few times and then stay down on a member of one of the port channels.  We lost communication to many of the servers during this time.  I then disabled the member port in UCS manager - after this normal communication seemed to resume.

Has anyone seen this behavior?

Is it better to be at a single connection?

Thanks in advance

3 REPLIES
Cisco Employee

6120 Port-Channel

It's definatley preferred to use a port-channel, as the pinning is directed to the logical port channel and then hashed out the appropriate link.  If you use individual links, UCS will still be able to re-pinned veth traffic from a failed link to a remaining operational one.

On the N5K's, have you configured the FI connections with port fast?  Since there's no STP on UCS (on the default End Host mode) you can safely enable this to prevent any upstream STP changes from impacting your uplinks.

On the N5K add "spanning-tree port type edge trunk" to the Port Channel interface.

What do the N5K logs show as the reason for the member interface going up/down? 

Is the flapping only on one interface between one of the  FI's only and the N5K?

If the flapping is isolated to one port between the FI and N5K, it could be a faulty transceiver and/or cable.  Might want to swap them out in this case.

If you think it's LACP related, then the logs should give us more information.

Regards,

Robert

New Member

6120 Port-Channel

Robert -

Thanks for the quick response.

Here is the port-channel / interface config :

interface Ethernet1/11

  description **Connection to UCS-A**

  switchport mode trunk

  switchport trunk allowed vlan 17,20-23,40-42,257,300,316,318,322,1500

  channel-group 10 mode active

interface Ethernet1/12

  description **Connection to UCS-A**

  switchport mode trunk

  switchport trunk allowed vlan 17,20-23,40-42,257,300,316,318,322,1500

  channel-group 10 mode active

interface port-channel10

  description **Connection to UCS-A**

  switchport mode trunk

  switchport trunk allowed vlan 17,20-23,40-42,257,300,316,318,322,1500

  spanning-tree port type edge trunk

The 5K shows the following at the affected time:

2011 Nov 30 10:54:32 600CenEast_N5048P_1 %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel10: Ethernet1/12 is down

2011 Nov 30 10:54:32 600CenEast_N5048P_1 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/12 is down (Link failure)

2011 Nov 30 10:54:44 600CenEast_N5048P_1 %ETHPORT-5-SPEED: Interface Ethernet1/12, operational speed changed to 10 Gbps

2011 Nov 30 10:54:44 600CenEast_N5048P_1 %ETHPORT-5-IF_DUPLEX: Interface Ethernet1/12, operational duplex mode changed to Full

2011 Nov 30 10:54:44 600CenEast_N5048P_1 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet1/12, operational Receive Flow Control state changed to off

2011 Nov 30 10:54:44 600CenEast_N5048P_1 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet1/12, operational Transmit Flow Control state changed to off

2011 Nov 30 10:54:47 600CenEast_N5048P_1 %ETH_PORT_CHANNEL-5-PORT_UP: port-channel10: Ethernet1/12 is up

2011 Nov 30 10:54:47 600CenEast_N5048P_1 %ETHPORT-5-IF_UP: Interface Ethernet1/12 is up in mode trunk

2011 Nov 30 10:54:50 600CenEast_N5048P_1 %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel10: Ethernet1/12 is down

2011 Nov 30 10:54:50 600CenEast_N5048P_1 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/12 is down (Link failure)

2011 Nov 30 10:54:51 600CenEast_N5048P_1 %ETHPORT-5-SPEED: Interface Ethernet1/12, operational speed changed to 10 Gbps

2011 Nov 30 10:54:51 600CenEast_N5048P_1 %ETHPORT-5-IF_DUPLEX: Interface Ethernet1/12, operational duplex mode changed to Full

2011 Nov 30 10:54:51 600CenEast_N5048P_1 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet1/12, operational Receive Flow Control state changed to off

2011 Nov 30 10:54:51 600CenEast_N5048P_1 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet1/12, operational Transmit Flow Control state changed to off

2011 Nov 30 10:54:54 600CenEast_N5048P_1 %ETH_PORT_CHANNEL-5-PORT_UP: port-channel10: Ethernet1/12 is up

2011 Nov 30 10:54:55 600CenEast_N5048P_1 %ETHPORT-5-IF_UP: Interface Ethernet1/12 is up in mode trunk

2011 Nov 30 10:54:59 600CenEast_N5048P_1 %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel10: Ethernet1/12 is down

2011 Nov 30 10:54:59 600CenEast_N5048P_1 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/12 is down (Link failure)

2011 Nov 30 10:55:00 600CenEast_N5048P_1 %ETHPORT-5-SPEED: Interface Ethernet1/12, operational speed changed to 10 Gbps

2011 Nov 30 10:55:00 600CenEast_N5048P_1 %ETHPORT-5-IF_DUPLEX: Interface Ethernet1/12, operational duplex mode changed to Full

2011 Nov 30 10:55:00 600CenEast_N5048P_1 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet1/12, operational Receive Flow Control state changed to off

2011 Nov 30 10:55:00 600CenEast_N5048P_1 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet1/12, operational Transmit Flow Control state changed to off

2011 Nov 30 10:55:03 600CenEast_N5048P_1 %ETH_PORT_CHANNEL-5-PORT_UP: port-channel10: Ethernet1/12 is up

2011 Nov 30 10:55:04 600CenEast_N5048P_1 %ETHPORT-5-IF_UP: Interface Ethernet1/12 is up in mode trunk

2011 Nov 30 10:55:43 600CenEast_N5048P_1 %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel10: Ethernet1/12 is down

2011 Nov 30 10:55:43 600CenEast_N5048P_1 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/12 is down (Link failure)

2011 Nov 30 10:55:54 600CenEast_N5048P_1 %ETHPORT-5-SPEED: Interface Ethernet1/12, operational speed changed to 10 Gbps

2011 Nov 30 10:55:54 600CenEast_N5048P_1 %ETHPORT-5-IF_DUPLEX: Interface Ethernet1/12, operational duplex mode changed to Full

2011 Nov 30 10:55:54 600CenEast_N5048P_1 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet1/12, operational Receive Flow Control state changed to off

2011 Nov 30 10:55:54 600CenEast_N5048P_1 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet1/12, operational Transmit Flow Control state changed to off

2011 Nov 30 10:55:58 600CenEast_N5048P_1 %ETH_PORT_CHANNEL-5-PORT_UP: port-channel10: Ethernet1/12 is up

2011 Nov 30 10:55:58 600CenEast_N5048P_1 %ETHPORT-5-IF_UP: Interface Ethernet1/12 is up in mode trunk

2011 Nov 30 10:56:09 600CenEast_N5048P_1 %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel10: Ethernet1/12 is down

2011 Nov 30 10:56:09 600CenEast_N5048P_1 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/12 is down (Link failure)

2011 Nov 30 10:56:20 600CenEast_N5048P_1 %ETHPORT-5-SPEED: Interface Ethernet1/12, operational speed changed to 10 Gbps

2011 Nov 30 10:56:20 600CenEast_N5048P_1 %ETHPORT-5-IF_DUPLEX: Interface Ethernet1/12, operational duplex mode changed to Full

2011 Nov 30 10:56:20 600CenEast_N5048P_1 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet1/12, operational Receive Flow Control state changed to off

2011 Nov 30 10:56:20 600CenEast_N5048P_1 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet1/12, operational Transmit Flow Control state changed to off

2011 Nov 30 10:56:30 600CenEast_N5048P_1 %ETH_PORT_CHANNEL-4-PORT_INDIVIDUAL: port Ethernet1/12 is operationally individual

2011 Nov 30 10:56:30 600CenEast_N5048P_1 %ETHPORT-5-IF_UP: Interface Ethernet1/12 is up in mode trunk

2011 Nov 30 10:56:30 600CenEast_N5048P_1 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/12 is down (Link failure)

2011 Nov 30 10:56:30 600CenEast_N5048P_1 %ETH_PORT_CHANNEL-5-PORT_INDIVIDUAL_DOWN: individual port Ethernet1/12 is down

2011 Nov 30 10:56:49 600CenEast_N5048P_1 %ETHPORT-5-SPEED: Interface Ethernet1/12, operational speed changed to 10 Gbps

2011 Nov 30 10:56:49 600CenEast_N5048P_1 %ETHPORT-5-IF_DUPLEX: Interface Ethernet1/12, operational duplex mode changed to Full

2011 Nov 30 10:56:49 600CenEast_N5048P_1 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet1/12, operational Receive Flow Control state changed to off

2011 Nov 30 10:56:49 600CenEast_N5048P_1 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet1/12, operational Transmit Flow Control state changed to off

2011 Nov 30 10:56:52 600CenEast_N5048P_1 %ETH_PORT_CHANNEL-5-PORT_UP: port-channel10: Ethernet1/12 is up

2011 Nov 30 10:56:52 600CenEast_N5048P_1 %ETHPORT-5-IF_UP: Interface Ethernet1/12 is up in mode trunk

2011 Nov 30 10:56:53 600CenEast_N5048P_1 %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel10: Ethernet1/12 is down

2011 Nov 30 10:56:53 600CenEast_N5048P_1 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/12 is down (Link failure)

2011 Nov 30 10:57:04 600CenEast_N5048P_1 %ETHPORT-5-SPEED: Interface Ethernet1/12, operational speed changed to 10 Gbps

2011 Nov 30 10:57:04 600CenEast_N5048P_1 %ETHPORT-5-IF_DUPLEX: Interface Ethernet1/12, operational duplex mode changed to Full

2011 Nov 30 10:57:04 600CenEast_N5048P_1 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet1/12, operational Receive Flow Control state changed to off

2011 Nov 30 10:57:04 600CenEast_N5048P_1 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet1/12, operational Transmit Flow Control state changed to off

2011 Nov 30 10:57:14 600CenEast_N5048P_1 %ETH_PORT_CHANNEL-4-PORT_INDIVIDUAL: port Ethernet1/12 is operationally individual

2011 Nov 30 10:57:14 600CenEast_N5048P_1 %ETHPORT-5-IF_UP: Interface Ethernet1/12 is up in mode trunk

2011 Nov 30 10:57:14 600CenEast_N5048P_1 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/12 is down (Link failure)

2011 Nov 30 10:57:14 600CenEast_N5048P_1 %ETH_PORT_CHANNEL-5-PORT_INDIVIDUAL_DOWN: individual port Ethernet1/12 is down

2011 Nov 30 10:57:47 600CenEast_N5048P_1 %ETHPORT-5-SPEED: Interface Ethernet1/12, operational speed changed to 10 Gbps

2011 Nov 30 10:57:47 600CenEast_N5048P_1 %ETHPORT-5-IF_DUPLEX: Interface Ethernet1/12, operational duplex mode changed to Full

2011 Nov 30 10:57:47 600CenEast_N5048P_1 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet1/12, operational Receive Flow Control state changed to off

2011 Nov 30 10:57:47 600CenEast_N5048P_1 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet1/12, operational Transmit Flow Control state changed to off

2011 Nov 30 10:57:50 600CenEast_N5048P_1 %ETH_PORT_CHANNEL-5-PORT_UP: port-channel10: Ethernet1/12 is up

2011 Nov 30 10:57:50 600CenEast_N5048P_1 %ETHPORT-5-IF_UP: Interface Ethernet1/12 is up in mode trunk

2011 Nov 30 11:25:56 600CenEast_N5048P_1 %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel10: Ethernet1/12 is down

2011 Nov 30 11:25:56 600CenEast_N5048P_1 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/12 is down (Link failure)

2011 Nov 30 11:25:57 600CenEast_N5048P_1 %ETHPORT-5-SPEED: Interface Ethernet1/12, operational speed changed to 10 Gbps

2011 Nov 30 11:25:57 600CenEast_N5048P_1 %ETHPORT-5-IF_DUPLEX: Interface Ethernet1/12, operational duplex mode changed to Full

2011 Nov 30 11:25:57 600CenEast_N5048P_1 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet1/12, operational Receive Flow Control state changed to off

2011 Nov 30 11:25:57 600CenEast_N5048P_1 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet1/12, operational Transmit Flow Control state changed to off

2011 Nov 30 11:26:07 600CenEast_N5048P_1 %ETH_PORT_CHANNEL-4-PORT_INDIVIDUAL: port Ethernet1/12 is operationally individual

2011 Nov 30 11:26:07 600CenEast_N5048P_1 %ETHPORT-5-IF_UP: Interface Ethernet1/12 is up in mode trunk

2011 Nov 30 11:26:07 600CenEast_N5048P_1 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/12 is down (Link failure)

2011 Nov 30 11:26:07 600CenEast_N5048P_1 %ETH_PORT_CHANNEL-5-PORT_INDIVIDUAL_DOWN: individual port Ethernet1/12 is down

As soon as I disabled the down port-channel member, traffic was again working properly. We are using a H10GB-CU1M cable for the connection, which we have had go bad before on other systems.  Just trying to figure out why this cause such an outage for us.

Thanks

Cisco Employee

6120 Port-Channel

If you have a "supsected" faulty cable, its hard to determine the implication of it failing.  If the link is hard down, traffic is repinned, but if its partially down to the point where LACP doesn't deem it down/failed, then traffic could be potentially black holed.

1. Enable port fast as recommended

2. Swap out that twinax cable.

Robert

1061
Views
0
Helpful
3
Replies
CreatePlease to create content