Buy or Renew
Buy or Renew
Cisco + Splunk: It’s a new day for your data. Learn more.
 
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
cancel
Start a conversation
1466
Views
0
Helpful
3
Replies

6120 Port-Channel

gsalisbu
Level 1
Level 1

‎11-30-2011 09:36 AM - edited ‎03-01-2019 10:10 AM

Hello -

We migrated from a 6120HA pair with a single connection between 6120A and a nexus 5020 and a single connection between 6120B and our other nexus 5020 to a port-channel between each of our FIs and 5020s (staight LACP, no vpc on nexus side).  The nexus switches are connected to each other with a 2x10gb Vpc peer link.  They are connected with a vpc to the 6509 VSS system at our core.

Each UCS chassis is doubly connected to its respective side of the FI.  FI A is connected strictly to the left side FEXs and FI B to the right side FEXs.

This morning we had a connection flap up and down a few times and then stay down on a member of one of the port channels.  We lost communication to many of the servers during this time.  I then disabled the member port in UCS manager - after this normal communication seemed to resume.

Has anyone seen this behavior?

Is it better to be at a single connection?

Thanks in advance

0 Helpful
3 Replies 3

Robert Burns
Cisco Employee
Cisco Employee

‎11-30-2011 10:03 AM

It's definatley preferred to use a port-channel, as the pinning is directed to the logical port channel and then hashed out the appropriate link.  If you use individual links, UCS will still be able to re-pinned veth traffic from a failed link to a remaining operational one.

On the N5K's, have you configured the FI connections with port fast?  Since there's no STP on UCS (on the default End Host mode) you can safely enable this to prevent any upstream STP changes from impacting your uplinks.

On the N5K add "spanning-tree port type edge trunk" to the Port Channel interface.

What do the N5K logs show as the reason for the member interface going up/down? 

Is the flapping only on one interface between one of the  FI's only and the N5K?

If the flapping is isolated to one port between the FI and N5K, it could be a faulty transceiver and/or cable.  Might want to swap them out in this case.

If you think it's LACP related, then the logs should give us more information.

Regards,

Robert

0 Helpful

gsalisbu
Level 1
Level 1
In response to Robert Burns

‎11-30-2011 10:11 AM

Robert -

Thanks for the quick response.

Here is the port-channel / interface config :

interface Ethernet1/11

  description **Connection to UCS-A**

  switchport mode trunk

  switchport trunk allowed vlan 17,20-23,40-42,257,300,316,318,322,1500

  channel-group 10 mode active

interface Ethernet1/12

  description **Connection to UCS-A**

  switchport mode trunk

  switchport trunk allowed vlan 17,20-23,40-42,257,300,316,318,322,1500

  channel-group 10 mode active

interface port-channel10

  description **Connection to UCS-A**

  switchport mode trunk

  switchport trunk allowed vlan 17,20-23,40-42,257,300,316,318,322,1500

  spanning-tree port type edge trunk

The 5K shows the following at the affected time:

2011 Nov 30 10:54:32 600CenEast_N5048P_1 %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel10: Ethernet1/12 is down

2011 Nov 30 10:54:32 600CenEast_N5048P_1 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/12 is down (Link failure)

2011 Nov 30 10:54:44 600CenEast_N5048P_1 %ETHPORT-5-SPEED: Interface Ethernet1/12, operational speed changed to 10 Gbps

2011 Nov 30 10:54:44 600CenEast_N5048P_1 %ETHPORT-5-IF_DUPLEX: Interface Ethernet1/12, operational duplex mode changed to Full

2011 Nov 30 10:54:44 600CenEast_N5048P_1 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet1/12, operational Receive Flow Control state changed to off

2011 Nov 30 10:54:44 600CenEast_N5048P_1 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet1/12, operational Transmit Flow Control state changed to off

2011 Nov 30 10:54:47 600CenEast_N5048P_1 %ETH_PORT_CHANNEL-5-PORT_UP: port-channel10: Ethernet1/12 is up

2011 Nov 30 10:54:47 600CenEast_N5048P_1 %ETHPORT-5-IF_UP: Interface Ethernet1/12 is up in mode trunk

2011 Nov 30 10:54:50 600CenEast_N5048P_1 %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel10: Ethernet1/12 is down

2011 Nov 30 10:54:50 600CenEast_N5048P_1 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/12 is down (Link failure)

2011 Nov 30 10:54:51 600CenEast_N5048P_1 %ETHPORT-5-SPEED: Interface Ethernet1/12, operational speed changed to 10 Gbps

2011 Nov 30 10:54:51 600CenEast_N5048P_1 %ETHPORT-5-IF_DUPLEX: Interface Ethernet1/12, operational duplex mode changed to Full

2011 Nov 30 10:54:51 600CenEast_N5048P_1 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet1/12, operational Receive Flow Control state changed to off

2011 Nov 30 10:54:51 600CenEast_N5048P_1 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet1/12, operational Transmit Flow Control state changed to off

2011 Nov 30 10:54:54 600CenEast_N5048P_1 %ETH_PORT_CHANNEL-5-PORT_UP: port-channel10: Ethernet1/12 is up

2011 Nov 30 10:54:55 600CenEast_N5048P_1 %ETHPORT-5-IF_UP: Interface Ethernet1/12 is up in mode trunk

2011 Nov 30 10:54:59 600CenEast_N5048P_1 %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel10: Ethernet1/12 is down

2011 Nov 30 10:54:59 600CenEast_N5048P_1 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/12 is down (Link failure)

2011 Nov 30 10:55:00 600CenEast_N5048P_1 %ETHPORT-5-SPEED: Interface Ethernet1/12, operational speed changed to 10 Gbps

2011 Nov 30 10:55:00 600CenEast_N5048P_1 %ETHPORT-5-IF_DUPLEX: Interface Ethernet1/12, operational duplex mode changed to Full

2011 Nov 30 10:55:00 600CenEast_N5048P_1 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet1/12, operational Receive Flow Control state changed to off

2011 Nov 30 10:55:00 600CenEast_N5048P_1 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet1/12, operational Transmit Flow Control state changed to off

2011 Nov 30 10:55:03 600CenEast_N5048P_1 %ETH_PORT_CHANNEL-5-PORT_UP: port-channel10: Ethernet1/12 is up

2011 Nov 30 10:55:04 600CenEast_N5048P_1 %ETHPORT-5-IF_UP: Interface Ethernet1/12 is up in mode trunk

2011 Nov 30 10:55:43 600CenEast_N5048P_1 %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel10: Ethernet1/12 is down

2011 Nov 30 10:55:43 600CenEast_N5048P_1 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/12 is down (Link failure)

2011 Nov 30 10:55:54 600CenEast_N5048P_1 %ETHPORT-5-SPEED: Interface Ethernet1/12, operational speed changed to 10 Gbps

2011 Nov 30 10:55:54 600CenEast_N5048P_1 %ETHPORT-5-IF_DUPLEX: Interface Ethernet1/12, operational duplex mode changed to Full

2011 Nov 30 10:55:54 600CenEast_N5048P_1 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet1/12, operational Receive Flow Control state changed to off

2011 Nov 30 10:55:54 600CenEast_N5048P_1 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet1/12, operational Transmit Flow Control state changed to off

2011 Nov 30 10:55:58 600CenEast_N5048P_1 %ETH_PORT_CHANNEL-5-PORT_UP: port-channel10: Ethernet1/12 is up

2011 Nov 30 10:55:58 600CenEast_N5048P_1 %ETHPORT-5-IF_UP: Interface Ethernet1/12 is up in mode trunk

2011 Nov 30 10:56:09 600CenEast_N5048P_1 %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel10: Ethernet1/12 is down

2011 Nov 30 10:56:09 600CenEast_N5048P_1 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/12 is down (Link failure)

2011 Nov 30 10:56:20 600CenEast_N5048P_1 %ETHPORT-5-SPEED: Interface Ethernet1/12, operational speed changed to 10 Gbps

2011 Nov 30 10:56:20 600CenEast_N5048P_1 %ETHPORT-5-IF_DUPLEX: Interface Ethernet1/12, operational duplex mode changed to Full

2011 Nov 30 10:56:20 600CenEast_N5048P_1 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet1/12, operational Receive Flow Control state changed to off

2011 Nov 30 10:56:20 600CenEast_N5048P_1 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet1/12, operational Transmit Flow Control state changed to off

2011 Nov 30 10:56:30 600CenEast_N5048P_1 %ETH_PORT_CHANNEL-4-PORT_INDIVIDUAL: port Ethernet1/12 is operationally individual

2011 Nov 30 10:56:30 600CenEast_N5048P_1 %ETHPORT-5-IF_UP: Interface Ethernet1/12 is up in mode trunk

2011 Nov 30 10:56:30 600CenEast_N5048P_1 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/12 is down (Link failure)

2011 Nov 30 10:56:30 600CenEast_N5048P_1 %ETH_PORT_CHANNEL-5-PORT_INDIVIDUAL_DOWN: individual port Ethernet1/12 is down

2011 Nov 30 10:56:49 600CenEast_N5048P_1 %ETHPORT-5-SPEED: Interface Ethernet1/12, operational speed changed to 10 Gbps

2011 Nov 30 10:56:49 600CenEast_N5048P_1 %ETHPORT-5-IF_DUPLEX: Interface Ethernet1/12, operational duplex mode changed to Full

2011 Nov 30 10:56:49 600CenEast_N5048P_1 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet1/12, operational Receive Flow Control state changed to off

2011 Nov 30 10:56:49 600CenEast_N5048P_1 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet1/12, operational Transmit Flow Control state changed to off

2011 Nov 30 10:56:52 600CenEast_N5048P_1 %ETH_PORT_CHANNEL-5-PORT_UP: port-channel10: Ethernet1/12 is up

2011 Nov 30 10:56:52 600CenEast_N5048P_1 %ETHPORT-5-IF_UP: Interface Ethernet1/12 is up in mode trunk

2011 Nov 30 10:56:53 600CenEast_N5048P_1 %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel10: Ethernet1/12 is down

2011 Nov 30 10:56:53 600CenEast_N5048P_1 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/12 is down (Link failure)

2011 Nov 30 10:57:04 600CenEast_N5048P_1 %ETHPORT-5-SPEED: Interface Ethernet1/12, operational speed changed to 10 Gbps

2011 Nov 30 10:57:04 600CenEast_N5048P_1 %ETHPORT-5-IF_DUPLEX: Interface Ethernet1/12, operational duplex mode changed to Full

2011 Nov 30 10:57:04 600CenEast_N5048P_1 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet1/12, operational Receive Flow Control state changed to off

2011 Nov 30 10:57:04 600CenEast_N5048P_1 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet1/12, operational Transmit Flow Control state changed to off

2011 Nov 30 10:57:14 600CenEast_N5048P_1 %ETH_PORT_CHANNEL-4-PORT_INDIVIDUAL: port Ethernet1/12 is operationally individual

2011 Nov 30 10:57:14 600CenEast_N5048P_1 %ETHPORT-5-IF_UP: Interface Ethernet1/12 is up in mode trunk

2011 Nov 30 10:57:14 600CenEast_N5048P_1 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/12 is down (Link failure)

2011 Nov 30 10:57:14 600CenEast_N5048P_1 %ETH_PORT_CHANNEL-5-PORT_INDIVIDUAL_DOWN: individual port Ethernet1/12 is down

2011 Nov 30 10:57:47 600CenEast_N5048P_1 %ETHPORT-5-SPEED: Interface Ethernet1/12, operational speed changed to 10 Gbps

2011 Nov 30 10:57:47 600CenEast_N5048P_1 %ETHPORT-5-IF_DUPLEX: Interface Ethernet1/12, operational duplex mode changed to Full

2011 Nov 30 10:57:47 600CenEast_N5048P_1 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet1/12, operational Receive Flow Control state changed to off

2011 Nov 30 10:57:47 600CenEast_N5048P_1 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet1/12, operational Transmit Flow Control state changed to off

2011 Nov 30 10:57:50 600CenEast_N5048P_1 %ETH_PORT_CHANNEL-5-PORT_UP: port-channel10: Ethernet1/12 is up

2011 Nov 30 10:57:50 600CenEast_N5048P_1 %ETHPORT-5-IF_UP: Interface Ethernet1/12 is up in mode trunk

2011 Nov 30 11:25:56 600CenEast_N5048P_1 %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel10: Ethernet1/12 is down

2011 Nov 30 11:25:56 600CenEast_N5048P_1 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/12 is down (Link failure)

2011 Nov 30 11:25:57 600CenEast_N5048P_1 %ETHPORT-5-SPEED: Interface Ethernet1/12, operational speed changed to 10 Gbps

2011 Nov 30 11:25:57 600CenEast_N5048P_1 %ETHPORT-5-IF_DUPLEX: Interface Ethernet1/12, operational duplex mode changed to Full

2011 Nov 30 11:25:57 600CenEast_N5048P_1 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet1/12, operational Receive Flow Control state changed to off

2011 Nov 30 11:25:57 600CenEast_N5048P_1 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet1/12, operational Transmit Flow Control state changed to off

2011 Nov 30 11:26:07 600CenEast_N5048P_1 %ETH_PORT_CHANNEL-4-PORT_INDIVIDUAL: port Ethernet1/12 is operationally individual

2011 Nov 30 11:26:07 600CenEast_N5048P_1 %ETHPORT-5-IF_UP: Interface Ethernet1/12 is up in mode trunk

2011 Nov 30 11:26:07 600CenEast_N5048P_1 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/12 is down (Link failure)

2011 Nov 30 11:26:07 600CenEast_N5048P_1 %ETH_PORT_CHANNEL-5-PORT_INDIVIDUAL_DOWN: individual port Ethernet1/12 is down

As soon as I disabled the down port-channel member, traffic was again working properly. We are using a H10GB-CU1M cable for the connection, which we have had go bad before on other systems.  Just trying to figure out why this cause such an outage for us.

Thanks

0 Helpful

Robert Burns
Cisco Employee
Cisco Employee
In response to gsalisbu

‎11-30-2011 10:21 AM

If you have a "supsected" faulty cable, its hard to determine the implication of it failing.  If the link is hard down, traffic is repinned, but if its partially down to the point where LACP doesn't deem it down/failed, then traffic could be potentially black holed.

1. Enable port fast as recommended

2. Swap out that twinax cable.

Robert

0 Helpful
Customers Also Viewed These Support Documents

Review Cisco Networking products for a $25 gift card