Buy or Renew
Buy or Renew
Cisco + Splunk: It’s a new day for your data. Learn more.
 
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
cancel
Start a conversation
5254
Views
20
Helpful
19
Replies

Added switch to network and err-disabled part of the network, why?

Andy White
Level 3
Level 3

‎09-07-2013 05:56 AM - edited ‎03-07-2019 03:20 PM

Hello,

Still not sure how I did this even after looking at the log files which are below.  Anyway we have a LAN which has 2 Nortel 8600 Core switch which are trunked together (IST - Nortel Etherchannel).  Off each Nortel are etherchannels to 2 Cisco 3750 stacks where our VMware farm and Storage SANs are connected.

Nortel fault.JPG

I plugged the Cisco 3750 into the Nortel switch shown above (port 1/27) and at that point I was fine, but when I on to the Nortel GUI manager and selected port 1/27 and these 3 options, band ports 12,24 on the 2 Cisco stacks went into error disable:

  • Perform Tagging
  • Discard Untagged Frames
  • Loop Detection

I then clicked apply and that did the trick and all went down.  I have a feeling it was the Loop Detection that cause the issue but why?

Here are the logs and the Cisco switch config I added.

Did the 2 Cisco stacks see the mac from 2 loctions?  What is strange I've don this one before on the other Nortel Core and it was fine.  I hate those Nortels.

# Generated by Syslog Viewer

# Version 2013.1.0

# 06/09/2013 18:06:42

#

# Syslog Messages

# 06 September 2013 06:06 PM

Gi3/0/33"          SW_MATM-4-MACFLAP_NOTIF          320517892

"06/09/2013 15:43:44"          192.168.28.251          192.168.28.251          Error          "26108:   Interface GigabitEthernet1/0/24, changed state to down"          LINK-3-UPDOWN          320517882

"06/09/2013 15:43:44"          192.168.28.251          192.168.28.251          Error          "26107:   Interface Port-channel1, changed state to down"          LINK-3-UPDOWN          320517891

"06/09/2013 15:43:43"          192.168.28.251          192.168.28.251          Notice          "26103:   Line protocol on Interface GigabitEthernet1/0/12, changed state to down"          LINEPROTO-5-UPDOWN          320517883

"06/09/2013 15:43:43"          192.168.28.251          192.168.28.251          Notice          "26104:   Line protocol on Interface GigabitEthernet1/0/24, changed state to down"          LINEPROTO-5-UPDOWN          320517885

"06/09/2013 15:43:43"          192.168.28.251          192.168.28.251          Notice          "26105:   Line protocol on Interface Port-channel1, changed state to down"          LINEPROTO-5-UPDOWN          320517887

"06/09/2013 15:43:43"          192.168.28.251          192.168.28.251          Error          "26106:   Interface GigabitEthernet1/0/12, changed state to down"          LINK-3-UPDOWN          320517889

"06/09/2013 15:43:42"          192.168.28.251          192.168.28.251          Warning          "26101:   channel-misconfig (STP) error detected on Po1, putting Gi1/0/24 in err-disable state"          PM-4-ERR_DISABLE          320517888

"06/09/2013 15:43:42"          192.168.28.251          192.168.28.251          Warning          "26100:   channel-misconfig (STP) error detected on Po1, putting Gi1/0/12 in err-disable state"          PM-4-ERR_DISABLE          320517886

"06/09/2013 15:43:42"          192.168.28.251          192.168.28.251          Warning          "26099:   channel-misconfig (STP) error detected on Gi1/0/24, putting Gi1/0/24 in err-disable state"          PM-4-ERR_DISABLE          320517884

"06/09/2013 15:43:42"          192.168.28.251          192.168.28.251          Warning          "26098:   channel-misconfig (STP) error detected on Gi1/0/12, putting Gi1/0/12 in err-disable state"          PM-4-ERR_DISABLE          320517881

"06/09/2013 15:43:42"          192.168.28.251          192.168.28.251          Warning          "26102:   channel-misconfig (STP) error detected on Po1, putting Po1 in err-disable state"          PM-4-ERR_DISABLE          320517890

"06/09/2013 15:43:31"          192.168.28.250          192.168.28.250          Warning          "29404: 032029:   Native VLAN mismatch discovered on GigabitEthernet1/0/24 (23), with Helpdesk_Switch.CBSO GigabitEthernet1/0/12 (1)."          CDP-4-NATIVE_VLAN_MISMATCH          320517880

"06/09/2013 15:43:31"          192.168.28.250          192.168.28.250          Warning          "29405: 000045:   Native VLAN mismatch discovered on GigabitEthernet1/0/24 (23), with Helpdesk_Switch.CBSO GigabitEthernet1/0/12 (1). (ISCSI-3750-A-1)"          CDP-4-NATIVE_VLAN_MISMATCH          320517879

"06/09/2013 15:43:31"          192.168.28.251          192.168.28.251          Warning          "26097:   Native VLAN mismatch discovered on GigabitEthernet1/0/24 (23), with Helpdesk_Switch.CBSO GigabitEthernet1/0/12 (1)."          CDP-4-NATIVE_VLAN_MISMATCH          320517878

"06/09/2013 15:42:32"          192.168.28.250          192.168.28.250          Warning          "29403: 000044:   Native VLAN mismatch discovered on GigabitEthernet1/0/24 (23), with Helpdesk_Switch.CBSO GigabitEthernet1/0/12 (1). (ISCSI-3750-A-1)"          CDP-4-NATIVE_VLAN_MISMATCH          320517875

"06/09/2013 15:42:31"          192.168.28.251          192.168.28.251          Warning          "26096:   Native VLAN mismatch discovered on GigabitEthernet1/0/24 (23), with Helpdesk_Switch.CBSO GigabitEthernet1/0/12 (1)."          CDP-4-NATIVE_VLAN_MISMATCH          320517873

"06/09/2013 15:42:31"          192.168.28.250          192.168.28.250          Warning          "29402: 032028:   Native VLAN mismatch discovered on GigabitEthernet1/0/24 (23), with Helpdesk_Switch.CBSO GigabitEthernet1/0/12 (1)."          CDP-4-NATIVE_VLAN_MISMATCH          320517874

"06/09/2013 15:42:27"          192.168.28.250          192.168.28.250          Notice          "29401: 032027:   Line protocol on Interface Vlan23, changed state to down"          LINEPROTO-5-UPDOWN          320517872

"06/09/2013 15:42:26"          192.168.28.251          192.168.28.251          Notice          "26095:   Line protocol on Interface Vlan23, changed state to down"          LINEPROTO-5-UPDOWN          320517871

"06/09/2013 15:42:26"          192.168.28.251          192.168.28.251          Notice          "26094:   Line protocol on Interface Vlan1, changed state to down"          LINEPROTO-5-UPDOWN          320517870

"06/09/2013 15:42:26"          192.168.28.251          192.168.28.251          Critical          "26093:   Blocking Port-channel1 on VLAN0023. Inconsistent local vlan."          SPANTREE-2-BLOCK_PVID_LOCAL          320517869

"06/09/2013 15:42:26"          192.168.28.251          192.168.28.251          Critical          "26092:   Blocking Port-channel1 on VLAN0001. Inconsistent peer vlan."          SPANTREE-2-BLOCK_PVID_PEER          320517868

"06/09/2013 15:42:26"          192.168.28.251          192.168.28.251          Critical          "26091:   Received BPDU with inconsistent peer vlan id 1 on Port-channel1 VLAN23."          SPANTREE-2-RECV_PVID_ERR          320517867

"06/09/2013 15:42:26"          192.168.28.250          192.168.28.250          Critical          "29400: 032026:   Blocking Port-channel1 on VLAN0023. Inconsistent local vlan."          SPANTREE-2-BLOCK_PVID_LOCAL          320517866

"06/09/2013 15:42:26"          192.168.28.250          192.168.28.250          Critical          "29399: 032025:   Blocking Port-channel1 on VLAN0001. Inconsistent peer vlan."          SPANTREE-2-BLOCK_PVID_PEER          320517865

"06/09/2013 15:42:26"          192.168.28.250          192.168.28.250          Critical          "29398: 032024:   Received BPDU with inconsistent peer vlan id 1 on Port-channel1 VLAN23."          SPANTREE-2-RECV_PVID_ERR          320517864

"06/09/2013 15:41:32"          192.168.28.250          192.168.28.250          Warning          "29397: 000043:   Native VLAN mismatch discovered on GigabitEthernet1/0/24 (23), with Helpdesk_Switch.CBSO GigabitEthernet1/0/12 (1). (ISCSI-3750-A-1)"          CDP-4-NATIVE_VLAN_MISMATCH          320517861

"06/09/2013 15:41:31"          192.168.28.251          192.168.28.251          Warning          "26090:   Native VLAN mismatch discovered on GigabitEthernet1/0/24 (23), with Helpdesk_Switch.CBSO GigabitEthernet1/0/12 (1)."          CDP-4-NATIVE_VLAN_MISMATCH          320517860

"06/09/2013 15:41:31"          192.168.28.250          192.168.28.250          Warning          "29396: 032023:   Native VLAN mismatch discovered on GigabitEthernet1/0/24 (23), with Helpdesk_Switch.CBSO GigabitEthernet1/0/12 (1)."          CDP-4-NATIVE_VLAN_MISMATCH          320517859

"06/09/2013 15:40:31"          192.168.28.250          192.168.28.250          Warning          "29394: 032022:   Native VLAN mismatch discovered on GigabitEthernet1/0/24 (23), with Helpdesk_Switch.CBSO GigabitEthernet1/0/12 (1)."          CDP-4-NATIVE_VLAN_MISMATCH          320517856

"06/09/2013 15:40:31"          192.168.28.250          192.168.28.250          Warning          "29395: 000042:   Native VLAN mismatch discovered on GigabitEthernet1/0/24 (23), with Helpdesk_Switch.CBSO GigabitEthernet1/0/12 (1). (ISCSI-3750-A-1)"          CDP-4-NATIVE_VLAN_MISMATCH          320517855

"06/09/2013 15:40:31"          192.168.28.251          192.168.28.251          Warning          "26089:   Native VLAN mismatch discovered on GigabitEthernet1/0/24 (23), with Helpdesk_Switch.CBSO GigabitEthernet1/0/12 (1)."          CDP-4-NATIVE_VLAN_MISMATCH          320517854

From one of the heldesk cisco switch I added that caused the issue.

*Mar  1 00:29:09.961: %PARSER-5-CFGLOG_LOGGEDCMD: User:console  logged command:ip dhcp snooping vlan 80

*Mar  1 00:29:22.997: %PARSER-5-CFGLOG_LOGGEDCMD: User:console  logged command:ip dhcp snooping

*Mar  1 00:30:10.050: %SYS-5-CONFIG_I: Configured from console by console

*Mar  1 00:40:11.588: %SYS-5-CONFIG_I: Configured from console by console

*Mar  1 00:42:02.462: %PARSER-5-CFGLOG_LOGGEDCMD: User:console  logged command:ip default-gateway 192.168.23.253

*Mar  1 00:42:03.435: %SYS-5-CONFIG_I: Configured from console by console

*Mar  1 00:54:38.807: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/12, changed state to up

*Mar  1 00:54:40.828: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/12, changed state to up

*Mar  1 00:57:44.483: %SPANTREE-2-RECV_PVID_ERR: Received BPDU with inconsistent peer vlan id 23 on GigabitEthernet1/0/12 VLAN1.

*Mar  1 00:57:44.483: %SPANTREE-2-BLOCK_PVID_LOCAL: Blocking GigabitEthernet1/0/12 on VLAN0001. Inconsistent local vlan.

*Mar  1 00:59:11.969: %SPANTREE-2-UNBLOCK_CONSIST_PORT: Unblocking GigabitEthernet1/0/12 on VLAN0001. Port consistency restored.

*Mar  1 00:59:27.907: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/12, changed state to down

*Mar  1 00:59:28.922: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/12, changed state to down

*Mar  1 03:28:31.432: %SEC_LOGIN-5-LOGIN_SUCCESS: Login Success [user: Andyw] [Source: 0.0.0.0] [localport: 0] at 03:28:31 UTC Mon Mar 1 1993

Helpdesk_Switch#sh run

Building configuration...

Current configuration : 3563 bytes

!

version 12.2

no service pad

service timestamps debug datetime msec

service timestamps log datetime msec

no service password-encryption

!

hostname Helpdesk_Switch

!

boot-start-marker

boot-end-marker

!

logging console informational

enable secret 5 $1$gyXtMMoalpx1SAcS7T.

!

username ***

!

!

no aaa new-model

switch 1 provision ws-c3750g-12s

system mtu routing 1500

ip domain-name ***

!

!

ip dhcp snooping vlan 80

ip dhcp snooping

login on-failure log

login on-success log

!

!

!

!

!

!

archive

log config

  logging enable

  logging size 200

  notify syslog contenttype plaintext

  hidekeys

!

spanning-tree mode rapid-pvst

spanning-tree extend system-id

!

vlan internal allocation policy ascending

!

ip ssh version 2

!

!

interface GigabitEthernet1/0/1

switchport access vlan 80

switchport mode access

spanning-tree portfast

spanning-tree bpduguard enable

!

!

(removed)

!

!

interface GigabitEthernet1/0/12

description Trunk to Nortel

switchport trunk encapsulation dot1q

switchport mode trunk

ip arp inspection trust

ip dhcp snooping trust

!

interface Vlan1

no ip address

shutdown

!

interface Vlan23

ip address 192.168.23.9 255.255.255.0

!

ip default-gateway 192.168.23.253

ip classless

no ip http server

no ip http secure-server

!

logging trap notifications

logging source-interface Vlan23

logging 192.168.*.*

access-list 20 permit 192.168.*.* log

access-list 20 permit 192.168.*.* log

access-list 20 permit 192.168.*.* log

access-list 20 permit 192.168.*.* log

access-list 24 permit 192.168.*.*

snmp-server community *** RO 24

snmp-server host 192.168.*.* ***

!

!

line con 0

exec-timeout 15 0

logging synchronous

login local

line vty 0 4

access-class 20 in

exec-timeout 15 0

logging synchronous

login local

transport input ssh

line vty 5 15

access-class 20 in

exec-timeout 15 0

logging synchronous

login local

transport input ssh

!

ntp logging

ntp clock-period 36029310

ntp source Vlan23

ntp server 192.168.*.*

end

Helpdesk_Switch#

Labels:
0 Helpful
19 Replies 19

Peter Paluch
Cisco Employee
Cisco Employee
In response to Andy White

‎09-09-2013 08:29 AM

Andy and everyone,

I apologize for intruding into this thread... What caught my attention was actually the reason why the ports appear to get err-disabled:

"06/09/2013 15:43:42"          192.168.28.251           192.168.28.251          Warning          "26101:   channel-misconfig  (STP) error detected on Po1, putting Gi1/0/24 in err-disable  state"          PM-4-ERR_DISABLE          320517888

"06/09/2013  15:43:42"          192.168.28.251          192.168.28.251           Warning          "26100:   channel-misconfig (STP) error detected on  Po1, putting Gi1/0/12 in err-disable state"           PM-4-ERR_DISABLE          320517886

"06/09/2013 15:43:42"           192.168.28.251          192.168.28.251          Warning           "26099:   channel-misconfig (STP) error detected on Gi1/0/24, putting  Gi1/0/24 in err-disable state"          PM-4-ERR_DISABLE           320517884

"06/09/2013 15:43:42"          192.168.28.251           192.168.28.251          Warning          "26098:   channel-misconfig  (STP) error detected on Gi1/0/12, putting Gi1/0/12 in err-disable  state"          PM-4-ERR_DISABLE          320517881

"06/09/2013  15:43:42"          192.168.28.251          192.168.28.251           Warning          "26102:   channel-misconfig (STP) error detected on  Po1, putting Po1 in err-disable state"           PM-4-ERR_DISABLE          320517890

This would mean that the ports in the Po1 EtherChannel bundle were receiving conflicting BPDUs - BPDUs that were sourced from different ports, perhaps even indicated different costs or port priorities. At the very least, the BPDUs were either sourced from different MAC addresses or differed in their Sending Port IDs. This would be a strong indication that the ports on the opposite switch were not operating as an EtherChannel bundle - otherwise the BPDUs would be sourced from a single MAC address and would be identical.

Andy, are you using LACP to negotiate the EtherChannel creation, or are you using the on mode?

Best regards,

Peter

0 Helpful

InayathUlla Sharieff
Cisco Employee
Cisco Employee
In response to Peter Paluch

‎09-09-2013 10:03 AM

I agree with Peter.

Hence thats the reason I always recommend to disable the interface atleast at one end,  followed by Creating the port-channel first @ bot the side then assign the same to the interface and enable the interface. This will avoid all the unreliable circumstances and bring up the port-channel without any issues.

HTH

REgards

Inayath

Peter Paluch
Cisco Employee
Cisco Employee
In response to InayathUlla Sharieff

‎09-09-2013 10:11 AM

Inayath,

Thank you! Your suggestion is bullet-proof - it would work all the time. I also feel, however, that it could introduce outages to the network because of the obvious fact of deactivating ports. My personal suggestion is to always use negotiated EtherChannels, either via LACP or via PAgP. Until these protocols negotiate the creation of an EtherChannel, both switches will treat the physical ports as being individual, and neither the EtherChannel STP Misconfig Guard will trip (as in this case), nor switching loops will ensue. It shall not be necessary to shut down ports when using LACP/PAgP.

Would that be your opinion as well - to always go with LACP or PAgP instead of the static "on" mode, in which case it is not necessary to shutdown ports?

Best regards,

Peter

InayathUlla Sharieff
Cisco Employee
Cisco Employee
In response to Peter Paluch

‎09-09-2013 10:20 AM

100% Agree with you. Always use one of the negotiation protocol when running etherchanel either with LACP/PAGPrather than "on" mode. +5 for you :-)

Yes but you know I have seen many people missing this piece of information and getting into loops. Hence it has become a practise for me to suggest this to many of them which have saved there time ...again as you said this needs the downtime as they need to shut the interface for some time.

Regards

Inayath.

0 Helpful

Andy White
Level 3
Level 3
In response to Peter Paluch

‎09-09-2013 12:53 PM

Hi guys, really appreciate all you input, here is the port info for both etherchannels on both stacks:

Stack A

sh int trunk

Port        Mode             Encapsulation  Status        Native vlan

Po1         on               802.1q         trunking      23

Port        Vlans allowed on trunk

Po1         1-4094

Port        Vlans allowed and active in management domain

Po1         1,23,1017-1019

Port        Vlans in spanning tree forwarding state and not pruned

Po1         1,23,1017-1019

sh etherchannel sum

Group  Port-channel  Protocol    Ports

------+-------------+-----------+-----------------------------------------------

1      Po1(SU)          -        Gi1/0/12(P) Gi1/0/24(P)

interface GigabitEthernet1/0/12

description Dot1q Trunk to Nortel Core 1 port 1/45

switchport trunk encapsulation dot1q

switchport trunk native vlan 23

switchport mode trunk

flowcontrol receive on

channel-group 1 mode on

end

interface GigabitEthernet1/0/24

description Dot1q Trunk to Nortel Core 2 port 1/45

switchport trunk encapsulation dot1q

switchport trunk native vlan 23

switchport mode trunk

flowcontrol receive on

channel-group 1 mode on

end

Stack B

sh int trunk

Port        Mode             Encapsulation  Status        Native vlan

Po1         on               802.1q         trunking      23

Port        Vlans allowed on trunk

Po1         1-4094

Port        Vlans allowed and active in management domain

Po1         1,23,1018-1019

Port        Vlans in spanning tree forwarding state and not pruned

Po1         1,23,1018-1019

sh ether sum

Group  Port-channel  Protocol    Ports

------+-------------+-----------+-----------------------------------------------

1      Po1(SU)          -        Gi1/0/12(P) Gi1/0/24(P)

interface GigabitEthernet1/0/12

description Dot1q Trunk to Nortel Core 1 port 2/45

switchport trunk encapsulation dot1q

switchport trunk native vlan 23

switchport mode trunk

flowcontrol receive on

channel-group 1 mode on

end

interface GigabitEthernet1/0/24

description Dot1q Trunk to Nortel Core 2 port 2/45

switchport trunk encapsulation dot1q

switchport trunk native vlan 23

switchport mode trunk

flowcontrol receive on

channel-group 1 mode on

end

Thanks

0 Helpful
Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community:

Customers Also Viewed These Support Documents
Review Cisco Networking products for a $25 gift card