interface resets causes Port channel to flap nexus 5010

Unanswered Question
Feb 21st, 2012

Hi all,

I am having the issues explained below , I did not find any bug related to the below error messages , your help is appreciated.

Here is an example of what each switch logs when a server drops offline.  Sample logs taken between 5:32am and 5:35am on Feb 20.   This particular one was having problems all weekend.  Switch #1 encountered over 2000 interface resets.  The corresponding VPC port on Switch #2 only had 13 resets.

I’d be happy to start a web meeting session with someone to show you more if it helps.

NEXUS SWITCH #1

2012 Feb 20 05:32:09 q91-sw01-5010 %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel10: first operational port changed

from Ethernet1/10 to none

2012 Feb 20 05:32:09 q91-sw01-5010 %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel10: Ethernet1/10 is down

2012 Feb 20 05:32:09 q91-sw01-5010 %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel10: port-channel10 is down

2012 Feb 20 05:32:09 q91-sw01-5010 %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel10 is down (

No operational members)

2012 Feb 20 05:32:09 q91-sw01-5010 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/10 is down (Link failure)

2012 Feb 20 05:32:09 q91-sw01-5010 %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel10 is down (

No operational members)

2012 Feb 20 05:32:19 q91-sw01-5010 %ETHPORT-5-SPEED: Interface Ethernet1/10, operational speed changed to 10 Gbps

2012 Feb 20 05:32:19 q91-sw01-5010 %ETHPORT-5-IF_DUPLEX: Interface Ethernet1/10, operational duplex mode changed to

Full

2012 Feb 20 05:32:19 q91-sw01-5010 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet1/10, operational Receive Flow

Control state changed to off

2012 Feb 20 05:32:19 q91-sw01-5010 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet1/10, operational Transmit Flow

Control state changed to off

2012 Feb 20 05:32:19 q91-sw01-5010 %ETHPORT-5-SPEED: Interface port-channel10, operational speed changed to 10 Gbps

2012 Feb 20 05:32:19 q91-sw01-5010 %ETHPORT-5-IF_DUPLEX: Interface port-channel10, operational duplex mode changed

to Full

2012 Feb 20 05:32:19 q91-sw01-5010 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface port-channel10, operational Receive Flo

w Control state changed to off

2012 Feb 20 05:32:19 q91-sw01-5010 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface port-channel10, operational Transmit Fl

ow Control state changed to off

2012 Feb 20 05:32:20 q91-sw01-5010 %ETH_PORT_CHANNEL-5-PORT_UP: port-channel10: Ethernet1/10 is up

2012 Feb 20 05:32:20 q91-sw01-5010 %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel10: first operational port changed

from none to Ethernet1/10

2012 Feb 20 05:32:20 q91-sw01-5010 %ETHPORT-5-IF_UP: Interface Ethernet1/10 is up in mode trunk

2012 Feb 20 05:32:20 q91-sw01-5010 %ETHPORT-3-IF_ERROR_VLANS_SUSPENDED: VLANs 999 on Interface port-channel10 are b

eing suspended. (Reason: Vlan is not allowed on Peer-link)

2012 Feb 20 05:32:20 q91-sw01-5010 %ETHPORT-5-IF_UP: Interface port-channel10 is up in mode trunk

2012 Feb 20 05:33:22 q91-sw01-5010 %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel10: first operational port changed

from Ethernet1/10 to none

2012 Feb 20 05:33:22 q91-sw01-5010 %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel10: Ethernet1/10 is down

2012 Feb 20 05:33:22 q91-sw01-5010 %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel10: port-channel10 is down

2012 Feb 20 05:33:22 q91-sw01-5010 %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel10 is down (

No operational members)

2012 Feb 20 05:33:22 q91-sw01-5010 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/10 is down (Link failure)

2012 Feb 20 05:33:22 q91-sw01-5010 %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel10 is down (

No operational members)

2012 Feb 20 05:33:32 q91-sw01-5010 %ETHPORT-5-SPEED: Interface Ethernet1/10, operational speed changed to 10 Gbps

2012 Feb 20 05:33:32 q91-sw01-5010 %ETHPORT-5-IF_DUPLEX: Interface Ethernet1/10, operational duplex mode changed to

Full

2012 Feb 20 05:33:32 q91-sw01-5010 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet1/10, operational Receive Flow

Control state changed to off

2012 Feb 20 05:33:32 q91-sw01-5010 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet1/10, operational Transmit Flow

Control state changed to off

2012 Feb 20 05:33:32 q91-sw01-5010 %ETHPORT-5-SPEED: Interface port-channel10, operational speed changed to 10 Gbps

2012 Feb 20 05:33:32 q91-sw01-5010 %ETHPORT-5-IF_DUPLEX: Interface port-channel10, operational duplex mode changed

to Full

2012 Feb 20 05:33:32 q91-sw01-5010 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface port-channel10, operational Receive Flo

w Control state changed to off

2012 Feb 20 05:33:32 q91-sw01-5010 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface port-channel10, operational Transmit Fl

ow Control state changed to off

2012 Feb 20 05:33:33 q91-sw01-5010 %ETH_PORT_CHANNEL-5-PORT_UP: port-channel10: Ethernet1/10 is up

2012 Feb 20 05:33:33 q91-sw01-5010 %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel10: first operational port changed

from none to Ethernet1/10

2012 Feb 20 05:33:33 q91-sw01-5010 %ETHPORT-5-IF_UP: Interface Ethernet1/10 is up in mode trunk

2012 Feb 20 05:33:33 q91-sw01-5010 %ETHPORT-3-IF_ERROR_VLANS_SUSPENDED: VLANs 999 on Interface port-channel10 are b

eing suspended. (Reason: Vlan is not allowed on Peer-link)

2012 Feb 20 05:33:33 q91-sw01-5010 %ETHPORT-5-IF_UP: Interface port-channel10 is up in mode trunk

2012 Feb 20 05:34:35 q91-sw01-5010 %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel10: first operational port changed

from Ethernet1/10 to none

2012 Feb 20 05:34:35 q91-sw01-5010 %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel10: Ethernet1/10 is down

2012 Feb 20 05:34:35 q91-sw01-5010 %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel10: port-channel10 is down

2012 Feb 20 05:34:35 q91-sw01-5010 %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel10 is down (

No operational members)

2012 Feb 20 05:34:35 q91-sw01-5010 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/10 is down (Link failure)

2012 Feb 20 05:34:35 q91-sw01-5010 %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel10 is down (

No operational members)

2012 Feb 20 05:34:46 q91-sw01-5010 %ETHPORT-5-SPEED: Interface Ethernet1/10, operational speed changed to 10 Gbps

2012 Feb 20 05:34:46 q91-sw01-5010 %ETHPORT-5-IF_DUPLEX: Interface Ethernet1/10, operational duplex mode changed to

Full

2012 Feb 20 05:34:46 q91-sw01-5010 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet1/10, operational Receive Flow

Control state changed to off

2012 Feb 20 05:34:46 q91-sw01-5010 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet1/10, operational Transmit Flow

Control state changed to off

2012 Feb 20 05:34:46 q91-sw01-5010 %ETHPORT-5-SPEED: Interface port-channel10, operational speed changed to 10 Gbps

2012 Feb 20 05:34:46 q91-sw01-5010 %ETHPORT-5-IF_DUPLEX: Interface port-channel10, operational duplex mode changed

to Full

2012 Feb 20 05:34:46 q91-sw01-5010 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface port-channel10, operational Receive Flo

w Control state changed to off

2012 Feb 20 05:34:46 q91-sw01-5010 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface port-channel10, operational Transmit Fl

ow Control state changed to off

2012 Feb 20 05:34:46 q91-sw01-5010 %ETH_PORT_CHANNEL-5-PORT_UP: port-channel10: Ethernet1/10 is up

2012 Feb 20 05:34:46 q91-sw01-5010 %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel10: first operational port changed

from none to Ethernet1/10

2012 Feb 20 05:34:46 q91-sw01-5010 %ETHPORT-5-IF_UP: Interface Ethernet1/10 is up in mode trunk

2012 Feb 20 05:34:46 q91-sw01-5010 %ETHPORT-3-IF_ERROR_VLANS_SUSPENDED: VLANs 999 on Interface port-channel10 are b

eing suspended. (Reason: Vlan is not allowed on Peer-link)

2012 Feb 20 05:34:46 q91-sw01-5010 %ETHPORT-5-IF_UP: Interface port-channel10 is up in mode trunk

2012 Feb 20 05:35:48 q91-sw01-5010 %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel10: first operational port changed

from Ethernet1/10 to none

NEXUS SWITCH #2

2012 Feb 20 05:32:20 q91-sw02-5010 %ETHPORT-3-IF_ERROR_VLANS_REMOVED: VLANs 999 on Interface port-channel10 are rem

oved from suspended state.

2012 Feb 20 05:32:20 q91-sw02-5010 %ETHPORT-3-IF_ERROR_VLANS_SUSPENDED: VLANs 999 on Interface port-channel10 are b

eing suspended. (Reason: Vlan is not allowed on Peer-link)

2012 Feb 20 05:33:33 q91-sw02-5010 %ETHPORT-3-IF_ERROR_VLANS_REMOVED: VLANs 999 on Interface port-channel10 are rem

oved from suspended state.

2012 Feb 20 05:33:33 q91-sw02-5010 %ETHPORT-3-IF_ERROR_VLANS_SUSPENDED: VLANs 999 on Interface port-channel10 are b

eing suspended. (Reason: Vlan is not allowed on Peer-link)

2012 Feb 20 05:34:46 q91-sw02-5010 %ETHPORT-3-IF_ERROR_VLANS_REMOVED: VLANs 999 on Interface port-channel10 are rem

oved from suspended state.

2012 Feb 20 05:34:46 q91-sw02-5010 %ETHPORT-3-IF_ERROR_VLANS_SUSPENDED: VLANs 999 on Interface port-channel10 are b

eing suspended. (Reason: Vlan is not allowed on Peer-link)

2012 Feb 20 05:35:59 q91-sw02-5010 %ETHPORT-3-IF_ERROR_VLANS_REMOVED: VLANs 999 on Interface port-channel10 are rem

oved from suspended state.

2012 Feb 20 05:35:59 q91-sw02-5010 %ETHPORT-3-IF_ERROR_VLANS_SUSPENDED: VLANs 999 on Interface port-channel10 are b

eing suspended. (Reason: Vlan is not allowed on Peer-link)

I have this problem too.
0 votes
  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 0 (0 ratings)
Oleksandr Nesterov Wed, 02/22/2012 - 11:38

Hi Saleem

It seems like port-channel 10 has only one link ethernet1/10 - and it flaps. Most probable cause - cble defect or sfp module defect. Start troubleshooting from replacing them.

If po10 is peerlink, and vlan 999 is only allowed vlan,  it flaps -  secondary peer will suspend vlan because it's not allowed on peerlink.

But there might be other scenarios.

Try to replace mentioned parts and then check.

HTH,

Alex

Actions

This Discussion