Cisco Support Community
cancel
Showing results for 
Search instead for 
Did you mean: 
Announcements
Webcast-Catalyst9k
New Member

Added switch to network and err-disabled part of the network, why?

Hello,

Still not sure how I did this even after looking at the log files which are below.  Anyway we have a LAN which has 2 Nortel 8600 Core switch which are trunked together (IST - Nortel Etherchannel).  Off each Nortel are etherchannels to 2 Cisco 3750 stacks where our VMware farm and Storage SANs are connected.

Nortel fault.JPG

I plugged the Cisco 3750 into the Nortel switch shown above (port 1/27) and at that point I was fine, but when I on to the Nortel GUI manager and selected port 1/27 and these 3 options, band ports 12,24 on the 2 Cisco stacks went into error disable:

  • Perform Tagging
  • Discard Untagged Frames
  • Loop Detection

I then clicked apply and that did the trick and all went down.  I have a feeling it was the Loop Detection that cause the issue but why?

Here are the logs and the Cisco switch config I added.

Did the 2 Cisco stacks see the mac from 2 loctions?  What is strange I've don this one before on the other Nortel Core and it was fine.  I hate those Nortels.

# Generated by Syslog Viewer

# Version 2013.1.0

# 06/09/2013 18:06:42

#

# Syslog Messages

# 06 September 2013 06:06 PM

Gi3/0/33"          SW_MATM-4-MACFLAP_NOTIF          320517892

"06/09/2013 15:43:44"          192.168.28.251          192.168.28.251          Error          "26108:   Interface GigabitEthernet1/0/24, changed state to down"          LINK-3-UPDOWN          320517882

"06/09/2013 15:43:44"          192.168.28.251          192.168.28.251          Error          "26107:   Interface Port-channel1, changed state to down"          LINK-3-UPDOWN          320517891

"06/09/2013 15:43:43"          192.168.28.251          192.168.28.251          Notice          "26103:   Line protocol on Interface GigabitEthernet1/0/12, changed state to down"          LINEPROTO-5-UPDOWN          320517883

"06/09/2013 15:43:43"          192.168.28.251          192.168.28.251          Notice          "26104:   Line protocol on Interface GigabitEthernet1/0/24, changed state to down"          LINEPROTO-5-UPDOWN          320517885

"06/09/2013 15:43:43"          192.168.28.251          192.168.28.251          Notice          "26105:   Line protocol on Interface Port-channel1, changed state to down"          LINEPROTO-5-UPDOWN          320517887

"06/09/2013 15:43:43"          192.168.28.251          192.168.28.251          Error          "26106:   Interface GigabitEthernet1/0/12, changed state to down"          LINK-3-UPDOWN          320517889

"06/09/2013 15:43:42"          192.168.28.251          192.168.28.251          Warning          "26101:   channel-misconfig (STP) error detected on Po1, putting Gi1/0/24 in err-disable state"          PM-4-ERR_DISABLE          320517888

"06/09/2013 15:43:42"          192.168.28.251          192.168.28.251          Warning          "26100:   channel-misconfig (STP) error detected on Po1, putting Gi1/0/12 in err-disable state"          PM-4-ERR_DISABLE          320517886

"06/09/2013 15:43:42"          192.168.28.251          192.168.28.251          Warning          "26099:   channel-misconfig (STP) error detected on Gi1/0/24, putting Gi1/0/24 in err-disable state"          PM-4-ERR_DISABLE          320517884

"06/09/2013 15:43:42"          192.168.28.251          192.168.28.251          Warning          "26098:   channel-misconfig (STP) error detected on Gi1/0/12, putting Gi1/0/12 in err-disable state"          PM-4-ERR_DISABLE          320517881

"06/09/2013 15:43:42"          192.168.28.251          192.168.28.251          Warning          "26102:   channel-misconfig (STP) error detected on Po1, putting Po1 in err-disable state"          PM-4-ERR_DISABLE          320517890

"06/09/2013 15:43:31"          192.168.28.250          192.168.28.250          Warning          "29404: 032029:   Native VLAN mismatch discovered on GigabitEthernet1/0/24 (23), with Helpdesk_Switch.CBSO GigabitEthernet1/0/12 (1)."          CDP-4-NATIVE_VLAN_MISMATCH          320517880

"06/09/2013 15:43:31"          192.168.28.250          192.168.28.250          Warning          "29405: 000045:   Native VLAN mismatch discovered on GigabitEthernet1/0/24 (23), with Helpdesk_Switch.CBSO GigabitEthernet1/0/12 (1). (ISCSI-3750-A-1)"          CDP-4-NATIVE_VLAN_MISMATCH          320517879

"06/09/2013 15:43:31"          192.168.28.251          192.168.28.251          Warning          "26097:   Native VLAN mismatch discovered on GigabitEthernet1/0/24 (23), with Helpdesk_Switch.CBSO GigabitEthernet1/0/12 (1)."          CDP-4-NATIVE_VLAN_MISMATCH          320517878

"06/09/2013 15:42:32"          192.168.28.250          192.168.28.250          Warning          "29403: 000044:   Native VLAN mismatch discovered on GigabitEthernet1/0/24 (23), with Helpdesk_Switch.CBSO GigabitEthernet1/0/12 (1). (ISCSI-3750-A-1)"          CDP-4-NATIVE_VLAN_MISMATCH          320517875

"06/09/2013 15:42:31"          192.168.28.251          192.168.28.251          Warning          "26096:   Native VLAN mismatch discovered on GigabitEthernet1/0/24 (23), with Helpdesk_Switch.CBSO GigabitEthernet1/0/12 (1)."          CDP-4-NATIVE_VLAN_MISMATCH          320517873

"06/09/2013 15:42:31"          192.168.28.250          192.168.28.250          Warning          "29402: 032028:   Native VLAN mismatch discovered on GigabitEthernet1/0/24 (23), with Helpdesk_Switch.CBSO GigabitEthernet1/0/12 (1)."          CDP-4-NATIVE_VLAN_MISMATCH          320517874

"06/09/2013 15:42:27"          192.168.28.250          192.168.28.250          Notice          "29401: 032027:   Line protocol on Interface Vlan23, changed state to down"          LINEPROTO-5-UPDOWN          320517872

"06/09/2013 15:42:26"          192.168.28.251          192.168.28.251          Notice          "26095:   Line protocol on Interface Vlan23, changed state to down"          LINEPROTO-5-UPDOWN          320517871

"06/09/2013 15:42:26"          192.168.28.251          192.168.28.251          Notice          "26094:   Line protocol on Interface Vlan1, changed state to down"          LINEPROTO-5-UPDOWN          320517870

"06/09/2013 15:42:26"          192.168.28.251          192.168.28.251          Critical          "26093:   Blocking Port-channel1 on VLAN0023. Inconsistent local vlan."          SPANTREE-2-BLOCK_PVID_LOCAL          320517869

"06/09/2013 15:42:26"          192.168.28.251          192.168.28.251          Critical          "26092:   Blocking Port-channel1 on VLAN0001. Inconsistent peer vlan."          SPANTREE-2-BLOCK_PVID_PEER          320517868

"06/09/2013 15:42:26"          192.168.28.251          192.168.28.251          Critical          "26091:   Received BPDU with inconsistent peer vlan id 1 on Port-channel1 VLAN23."          SPANTREE-2-RECV_PVID_ERR          320517867

"06/09/2013 15:42:26"          192.168.28.250          192.168.28.250          Critical          "29400: 032026:   Blocking Port-channel1 on VLAN0023. Inconsistent local vlan."          SPANTREE-2-BLOCK_PVID_LOCAL          320517866

"06/09/2013 15:42:26"          192.168.28.250          192.168.28.250          Critical          "29399: 032025:   Blocking Port-channel1 on VLAN0001. Inconsistent peer vlan."          SPANTREE-2-BLOCK_PVID_PEER          320517865

"06/09/2013 15:42:26"          192.168.28.250          192.168.28.250          Critical          "29398: 032024:   Received BPDU with inconsistent peer vlan id 1 on Port-channel1 VLAN23."          SPANTREE-2-RECV_PVID_ERR          320517864

"06/09/2013 15:41:32"          192.168.28.250          192.168.28.250          Warning          "29397: 000043:   Native VLAN mismatch discovered on GigabitEthernet1/0/24 (23), with Helpdesk_Switch.CBSO GigabitEthernet1/0/12 (1). (ISCSI-3750-A-1)"          CDP-4-NATIVE_VLAN_MISMATCH          320517861

"06/09/2013 15:41:31"          192.168.28.251          192.168.28.251          Warning          "26090:   Native VLAN mismatch discovered on GigabitEthernet1/0/24 (23), with Helpdesk_Switch.CBSO GigabitEthernet1/0/12 (1)."          CDP-4-NATIVE_VLAN_MISMATCH          320517860

"06/09/2013 15:41:31"          192.168.28.250          192.168.28.250          Warning          "29396: 032023:   Native VLAN mismatch discovered on GigabitEthernet1/0/24 (23), with Helpdesk_Switch.CBSO GigabitEthernet1/0/12 (1)."          CDP-4-NATIVE_VLAN_MISMATCH          320517859

"06/09/2013 15:40:31"          192.168.28.250          192.168.28.250          Warning          "29394: 032022:   Native VLAN mismatch discovered on GigabitEthernet1/0/24 (23), with Helpdesk_Switch.CBSO GigabitEthernet1/0/12 (1)."          CDP-4-NATIVE_VLAN_MISMATCH          320517856

"06/09/2013 15:40:31"          192.168.28.250          192.168.28.250          Warning          "29395: 000042:   Native VLAN mismatch discovered on GigabitEthernet1/0/24 (23), with Helpdesk_Switch.CBSO GigabitEthernet1/0/12 (1). (ISCSI-3750-A-1)"          CDP-4-NATIVE_VLAN_MISMATCH          320517855

"06/09/2013 15:40:31"          192.168.28.251          192.168.28.251          Warning          "26089:   Native VLAN mismatch discovered on GigabitEthernet1/0/24 (23), with Helpdesk_Switch.CBSO GigabitEthernet1/0/12 (1)."          CDP-4-NATIVE_VLAN_MISMATCH          320517854

From one of the heldesk cisco switch I added that caused the issue.

*Mar  1 00:29:09.961: %PARSER-5-CFGLOG_LOGGEDCMD: User:console  logged command:ip dhcp snooping vlan 80

*Mar  1 00:29:22.997: %PARSER-5-CFGLOG_LOGGEDCMD: User:console  logged command:ip dhcp snooping

*Mar  1 00:30:10.050: %SYS-5-CONFIG_I: Configured from console by console

*Mar  1 00:40:11.588: %SYS-5-CONFIG_I: Configured from console by console

*Mar  1 00:42:02.462: %PARSER-5-CFGLOG_LOGGEDCMD: User:console  logged command:ip default-gateway 192.168.23.253

*Mar  1 00:42:03.435: %SYS-5-CONFIG_I: Configured from console by console

*Mar  1 00:54:38.807: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/12, changed state to up

*Mar  1 00:54:40.828: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/12, changed state to up

*Mar  1 00:57:44.483: %SPANTREE-2-RECV_PVID_ERR: Received BPDU with inconsistent peer vlan id 23 on GigabitEthernet1/0/12 VLAN1.

*Mar  1 00:57:44.483: %SPANTREE-2-BLOCK_PVID_LOCAL: Blocking GigabitEthernet1/0/12 on VLAN0001. Inconsistent local vlan.

*Mar  1 00:59:11.969: %SPANTREE-2-UNBLOCK_CONSIST_PORT: Unblocking GigabitEthernet1/0/12 on VLAN0001. Port consistency restored.

*Mar  1 00:59:27.907: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/12, changed state to down

*Mar  1 00:59:28.922: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/12, changed state to down

*Mar  1 03:28:31.432: %SEC_LOGIN-5-LOGIN_SUCCESS: Login Success [user: Andyw] [Source: 0.0.0.0] [localport: 0] at 03:28:31 UTC Mon Mar 1 1993

Helpdesk_Switch#sh run

Building configuration...

Current configuration : 3563 bytes

!

version 12.2

no service pad

service timestamps debug datetime msec

service timestamps log datetime msec

no service password-encryption

!

hostname Helpdesk_Switch

!

boot-start-marker

boot-end-marker

!

logging console informational

enable secret 5 $1$gyXtMMoalpx1SAcS7T.

!

username ***

!

!

no aaa new-model

switch 1 provision ws-c3750g-12s

system mtu routing 1500

ip domain-name ***

!

!

ip dhcp snooping vlan 80

ip dhcp snooping

login on-failure log

login on-success log

!

!

!

!

!

!

archive

log config

  logging enable

  logging size 200

  notify syslog contenttype plaintext

  hidekeys

!

spanning-tree mode rapid-pvst

spanning-tree extend system-id

!

vlan internal allocation policy ascending

!

ip ssh version 2

!

!

interface GigabitEthernet1/0/1

switchport access vlan 80

switchport mode access

spanning-tree portfast

spanning-tree bpduguard enable

!

!

(removed)

!

!

interface GigabitEthernet1/0/12

description Trunk to Nortel

switchport trunk encapsulation dot1q

switchport mode trunk

ip arp inspection trust

ip dhcp snooping trust

!

interface Vlan1

no ip address

shutdown

!

interface Vlan23

ip address 192.168.23.9 255.255.255.0

!

ip default-gateway 192.168.23.253

ip classless

no ip http server

no ip http secure-server

!

logging trap notifications

logging source-interface Vlan23

logging 192.168.*.*

access-list 20 permit 192.168.*.* log

access-list 20 permit 192.168.*.* log

access-list 20 permit 192.168.*.* log

access-list 20 permit 192.168.*.* log

access-list 24 permit 192.168.*.*

snmp-server community *** RO 24

snmp-server host 192.168.*.* ***

!

!

line con 0

exec-timeout 15 0

logging synchronous

login local

line vty 0 4

access-class 20 in

exec-timeout 15 0

logging synchronous

login local

transport input ssh

line vty 5 15

access-class 20 in

exec-timeout 15 0

logging synchronous

login local

transport input ssh

!

ntp logging

ntp clock-period 36029310

ntp source Vlan23

ntp server 192.168.*.*

end

Helpdesk_Switch#

19 REPLIES

Added switch to network and err-disabled part of the network, wh

Ah STP issues, get rid of them Nortels lol. 

Ok so some things to check here.  Make sure that all switches are running same STP mode from your config Help Desk is running rstp so make sure all the others are running the same.  Also do you have the native VLAN configured as a different VLAN on the Cisco Stacks and Nortel vs the Help Desk switch (Cisco Help Desk switch looks like is using the default VLAN1 as the native VLAN but others aren't ?  Make sure that matches.

Re: Added switch to network and err-disabled part of the network

I'm no nortel expert but it looks like the nortels are expecting native vlan 23 whereas the Cisco is configured as native vlan 1 on the trunk port. If that's correct you need to update the trunk port on the ciscos to switchport trunk native vlan23.

Sent from Cisco Technical Support iPad App

New Member

Added switch to network and err-disabled part of the network, wh

It is using Native VLAN 1, but so is the other Cisco switch I installed on the other Nortel core that didn't cause any issues.  Why would a native vlan mismatch cause an issue on the 2 separate Cisco switch stacks and cause them to go into err-disable?

I've had native vlan mismatches before and it never caused the ports to shutdown or a STP mismatch.

Thanks

Re: Added switch to network and err-disabled part of the network

Hi Andy,

the Cisco per-VLAN STP modes send so-called SSTP BPDUs to the destination address 01-00-0C-CC-CC-CD.

Third party switches normally don't recognize them as BPDUs (IEEE destination address is 01-80-C2-00-00-00) but as the group bit is set, treat them as multicast and flood them. So the Nortel switches act as virtual trunks for the per-VLAN ST instances between the c3750 stacks and the helpdesk switch.

Anyway we have a LAN which has 2 Nortel 8600 Core switch which are trunked together (IST - Nortel Etherchannel).

Are you saying the mode in this part is MSTP?

If so, the the receipt of SSTP BPDUs on switchports running MSTP will result in ST inconsistent state and errdisable the port for ~2 minutes, then the same procedure starts again.

If the rest of the multi-vendor LAN is running MSTP, I'd recommend to configure MSTP on the helpdesk switch as well.

Useful link: Troubleshooting Spanning Tree PVID- and Type-Inconsistencies

Hope that helps

Rolf

Cisco Employee

Re: Added switch to network and err-disabled part of the network

Hi Rolf,

Good day.

If so, the the receipt of SSTP BPDUs on switchports running MSTP will result in ST inconsistent state and errdisable the port for ~2 minutes, then the same procedure starts again.

I'm not sure if this is being stated with specifically Nortel switches in mind, but the above is not true for Cisco switches. Cisco switches running MST can interoperate with other switches running PVST+ (or Rapid-PVST+) without any problems. It does get a little complicated, I won't deny that, but in most situations it works fine without any problems.

Regards,

Aninda

Re: Added switch to network and err-disabled part of the network

Hi Aninda,

thanks for joining us.

Cisco switches running MST can interoperate with other switches running PVST+ (or Rapid-PVST+) without any problems.

Of course that's correct, but here we could have a very special scenario where some of the ports receive SSTP- and MST BPDUs at the same time so the switch cannot figure out the correct mode.

Anyway, MSTP is obviously not the point here. I've been reading a Nortel Layer-2 configuration guide and found that "IST" stands for "InterSwitch Trunking" (link aggregation).

Best regards

Rolf

Re: Added switch to network and err-disabled part of the network

Here are couple of links to explain more in detail:

Basically with the mismatch you are adding inconsistency to the ports.  If you run "show spanning-tree       inconsistentports" you will see the ports will show up there. 

Not sure about the other switch are you pruning VLAN1, do you may be have loopguard setup?  Also then there an issue of inter operability between Nortel and Cisco's implementation of STP. 

https://supportforums.cisco.com/thread/2151150

http://www.cisco.com/en/US/tech/tk389/tk621/technologies_tech_note09186a00801d11a0.shtml

http://www.informit.com/library/content.aspx?b=CCNP_Studies_Switching&seqNum=25

Hope this helps a bit.

Note:  It is a good idea I believe to keep things simple and standard.  Such as matching Native VLAN's, port configurations, port channel configs, STP modes etc.  Will make troubleshooting much easier.

Cisco Employee

Re: Added switch to network and err-disabled part of the network

Hi Andy,

Both Rolf and Mohammad have given very good information here. It is quite important to understand how the Cisco proprietary PVST+ (or Rapid-PVST+) really works and more interestingly, how they interoperate with third party switches - in this case, Nortel switches.

Both PVST+ and Rapid-PVST+ send their BPDUs using a destination mac-address which is Cisco proprietary and which is not understood by most third party switches (there are some third party switches which have the capability to turn on a feature to understand these BPDUs). This creates a problem with Nortel switches in the middle because they will not understand BPDUs for VLANs 2 and above (VLAN 1 would work fine because an IEEE BPDU would be sent out for this as well) and they simply flood these out without processing them.

Any Cisco switches behind the Nortels would now be receiving these BPDUs and will be processing them.

A snippet of the errors you saw:

*Mar  1 00:57:44.483: %SPANTREE-2-RECV_PVID_ERR: Received BPDU with inconsistent peer vlan id 23 on GigabitEthernet1/0/12 VLAN1.

*Mar  1 00:57:44.483: %SPANTREE-2-BLOCK_PVID_LOCAL: Blocking GigabitEthernet1/0/12 on VLAN0001. Inconsistent local vlan.

This indicates that the local switch received a BPDU which was untagged, however, the VLAN ID in the TLV of the BDPU was set to 23. This is being compared to the native VLAN on the local switch's receiving interface which is 1. Because of the mismatch, spanning-tree blocks the interfaces for both the VLANs.

From your topology, you have two Cisco switches on top - can you post the show run interface output for all the interfaces from these switches that go to the Nortel switches please?

Regards,

Aninda

Cisco Employee

Re: Added switch to network and err-disabled part of the network

I've had native vlan mismatches before and it never caused the ports to shutdown or a STP mismatch.

Native VLAN mismatches would not cause a port to be shut down, but it would most certainly cause spanning-tree to block the mismatched VLAN pair for that interface.

There was quite an interesting discussion on how this happens sometime back -

https://supportforums.cisco.com/message/4001268#4001268

Please feel free to ask further!

Regards,

Aninda

New Member

Re: Added switch to network and err-disabled part of the network

Thanks guys for you info, I have a couple of questions.  What confuses me I'm already done this and it is production.

I configured the Cisco switch first and plugged in and plugged in and all was ok, then I went on to the Nortel manager and selected the 4 options and hit apply and then the issue started.

Nortel fault 2.JPG

The Cisco switch I added must be using native VLAN 1 but so is the other Cisco switch I added that worked

1.) So in summary the Nortel switch didn't know what to do with the this BPDU and flooded it using multicast so both Cisco stacks (virtual server farm) received it and saw it as a native vlan mismatch between vlan 1 and 23 (should of been 23?) and shutdown the etherchannes which were ports 12 and 24?

2.) Is a native VLAN mismatch's default behavior to err-disable the port and shut it down then?

3.) When this happened I pulled the Cisco switch I added off the network then I consoled onto one of the Cisco 3750 stacks where the etherchannel was down (ports 12 & 24) and did a 'shut' and 'no shut' on the ports to find they err-disabled immediately again, why?

4.) I then pulled the etherchannel cables out (trunk ports 12 & 24) so there was no connection to the Nortels and did a 'shut' then 'no shut' and the ports err-disabled again, why would they do they if they we no longer connect to anything?

5.)  I had to reload the switch (stack) to get it all working.  I guess it clear the mac address table, is there a 'soft' way to clear the table?

Thanks in advance

Re: Added switch to network and err-disabled part of the network

Andy,

I've been reading some Nortel documentation to understand their Layer-2 concepts and I have to say that there are some things I still don't understand entirely.

Nortel Switches also have some proprietary STP enhencaments, like STG (ST groups). STG1 BPDUs are normal IEEE-untagged BPDUs, STG 2-64 send tagged BPDUs with IEEE-format. So the only common ST is between Cisco VLAN1 and Nortel STG1, the rest (Cisco PVST-instances and Nortel STG >1) are partitioned. I wonder what a Catalyst Switch does when it receives a tagged IEEE BPDU ...

Could you tell us if you use STG others than STG1?

The screenshot  above shows the trunkport configuration to the added helpdesk-switch, right?

The "DiscardUntaggedFrames" option is enabled, so I'd expect that received control traffic like CDP will be dropped here.

However, we can find syslog messages like:

"06/09/2013 15:40:31"          192.168.28.250          192.168.28.250          Warning          "29395: 000042:   Native VLAN mismatch discovered on GigabitEthernet1/0/24 (23), with Helpdesk_Switch.CBSO GigabitEthernet1/0/12 (1). (ISCSI-3750-A-1)"          CDP-4-NATIVE_VLAN_MISMATCH

Almost all Layer-2 control protocols are always send untagged, even if VLAN1 is manually pruned or another VLAN is configured as Native-VLAN. So how could that happen ...

The very first syslog message is  a SW_MATM-4-MACFLAP notification, could you tell us if that MAC-address came form the helpdesk-switch or from the original LAN (Nortel, c3750 stacks)?

After that, member-ports of the etherchannel went down:

"06/09/2013 15:43:44"          192.168.28.251          192.168.28.251          Error          "26108:   Interface GigabitEthernet1/0/24, changed state to down"          LINK-3-UPDOWN          320517882

This could have been caused by the Nortel's LoopDetection Feature:

On a per-port basis, the Loop Detection feature detects MAC addresses that are looping from one port to other ports. After a loop is detected, the port on which the MAC addresses were learned is disabled. Additionally, if a MAC address is found to loop, the MAC address is disabled for that VLAN.

Perhaps analysing the syslogs can help us reconstructing what happend and what was cause and effect.

Generally, a port is put in ErrDisabled to avoid even worse situations. I think there must have been more than just a mismatch of native VLANs .

Best regards

Rolf

[edit]

I just noticed that the syslog messages in the original post are not in ascending timestamps-order.

So the MAC flap was not the first event ...

New Member

Re: Added switch to network and err-disabled part of the network

I'd like to lab this topology, but don't have any spare Nortel switches, I have spare Cisco switches, is there a way to cause this behavour between 2 Cisco switches?

I just trunked 2 switches and the native vlan's were different and all was good still.

This still confuses me, I guess a multi vendor LAN is every mans nightmare, we hope to replace the 2 Nortels witch 2 x 3850x's

Any ideas on these?

1.) So in summary the Nortel switch didn't know what to do with the this BPDU and flooded it using multicast so both Cisco stacks (virtual server farm) recieved it and saw it as a native vlan mismatch between vlan 1 and 23 (should of been 23?) and shutdown the etherchannes which were ports 12 and 24?

2.) Is a native VLAN mismatch's default behavior to err-disable the port and shut it down then?

3.) When this happened I pulled the Cisco switch I added off the network then I consoled onto one of the Cisco 3750 stacks where the etherchannel was down (ports 12 & 24) and did a 'shut' and 'no shut' on the ports to find they err-disabled immediately again, why?

4.) I then pulled the etherchannel cables out (trunk ports 12 & 24) so there was no connection to the Nortels and did a 'shut' then 'no shut' and the ports err-disabled again, why would they do they if they we no longer connect to anything?

5.)  I had to reload the switch (stack) to get it all working.  I guess it clear the mac address table, is there a 'soft' way to clear the table?

Re: Added switch to network and err-disabled part of the network

1) The Native VLAN mismatch made the Catalysts block VLAN 1 and 23 on Po1

15:42:26 SPANTREE-2-RECV_PVID_ERR (Result: VLAN 1 and 23 are blocked on Po1)

Since Po1 was the only ST active port  for VLANs 1 and 23, the SVIs went down at the same moment.

15:42:26 LINEPROTO-5-UPDOWN (Interface VLAN 1)

15:42:26 LINEPROTO-5-UPDOWN (Interface VLAN 23)

1 1/2 minutes later, the etherchannel was errdisabled because a EC misconfiguration was assumed.

15:43:42 PM-4-ERR_DISABLE (Po1, channel-misconfig STP)

2) No, the mismatched VLANs will be blocked but the ports are not errdisabled. Technically it was an etherchannel/STP protection mechanism that put the EC in errdisable.

3-4) Perhaps the Nortels unbundeld the channel. Another interesting link: Errdisable Port State Recovery on the Cisco IOS Platforms

Recover a Port from Errdisabled State

After you discover why the ports were disabled, fix the root problem. The fix depends on the triggering problem. (...)

I know, not really a satisfactroy reply, but the best I can offer.

Best regards

Rolf

P.S.: Setting up this scenario in a  lab would be very interesting, but I think you'll need some third-party switches.

New Member

Re: Added switch to network and err-disabled part of the network

Thanks for the time you have spent on this issue.

As you know I pulled te trunk cables out of the 2 x 3750 stacks and tried a 'shut' & 'no shut' on ports 12 & 24 and it still err-disabled, rather than reload the 2 switches which fixed the issue, could I of don't something less drastic?  I can only think the reload cleared the Mac/CAM table which helped.

So in brief this was down to the Nortel's not understanding the BPDU sent from the helpdesk switch and flooding it so the 2 Cisco stacks received on their po1 etherchannels and as it was their default behavior to shut down because they expected a Native VLAN of 1, but got 23?

Regards

Cisco Employee

Re: Added switch to network and err-disabled part of the network

Andy and everyone,

I apologize for intruding into this thread... What caught my attention was actually the reason why the ports appear to get err-disabled:

"06/09/2013 15:43:42"          192.168.28.251           192.168.28.251          Warning          "26101:   channel-misconfig  (STP) error detected on Po1, putting Gi1/0/24 in err-disable  state"          PM-4-ERR_DISABLE          320517888

"06/09/2013  15:43:42"          192.168.28.251          192.168.28.251           Warning          "26100:   channel-misconfig (STP) error detected on  Po1, putting Gi1/0/12 in err-disable state"           PM-4-ERR_DISABLE          320517886

"06/09/2013 15:43:42"           192.168.28.251          192.168.28.251          Warning           "26099:   channel-misconfig (STP) error detected on Gi1/0/24, putting  Gi1/0/24 in err-disable state"          PM-4-ERR_DISABLE           320517884

"06/09/2013 15:43:42"          192.168.28.251           192.168.28.251          Warning          "26098:   channel-misconfig  (STP) error detected on Gi1/0/12, putting Gi1/0/12 in err-disable  state"          PM-4-ERR_DISABLE          320517881

"06/09/2013  15:43:42"          192.168.28.251          192.168.28.251           Warning          "26102:   channel-misconfig (STP) error detected on  Po1, putting Po1 in err-disable state"           PM-4-ERR_DISABLE          320517890

This would mean that the ports in the Po1 EtherChannel bundle were receiving conflicting BPDUs - BPDUs that were sourced from different ports, perhaps even indicated different costs or port priorities. At the very least, the BPDUs were either sourced from different MAC addresses or differed in their Sending Port IDs. This would be a strong indication that the ports on the opposite switch were not operating as an EtherChannel bundle - otherwise the BPDUs would be sourced from a single MAC address and would be identical.

Andy, are you using LACP to negotiate the EtherChannel creation, or are you using the on mode?

Best regards,

Peter

Cisco Employee

Added switch to network and err-disabled part of the network, wh

I agree with Peter.

Hence thats the reason I always recommend to disable the interface atleast at one end,  followed by Creating the port-channel first @ bot the side then assign the same to the interface and enable the interface. This will avoid all the unreliable circumstances and bring up the port-channel without any issues.

HTH

REgards

Inayath

Cisco Employee

Added switch to network and err-disabled part of the network, wh

Inayath,

Thank you! Your suggestion is bullet-proof - it would work all the time. I also feel, however, that it could introduce outages to the network because of the obvious fact of deactivating ports. My personal suggestion is to always use negotiated EtherChannels, either via LACP or via PAgP. Until these protocols negotiate the creation of an EtherChannel, both switches will treat the physical ports as being individual, and neither the EtherChannel STP Misconfig Guard will trip (as in this case), nor switching loops will ensue. It shall not be necessary to shut down ports when using LACP/PAgP.

Would that be your opinion as well - to always go with LACP or PAgP instead of the static "on" mode, in which case it is not necessary to shutdown ports?

Best regards,

Peter

Cisco Employee

Added switch to network and err-disabled part of the network, wh

100% Agree with you. Always use one of the negotiation protocol when running etherchanel either with LACP/PAGPrather than "on" mode. +5 for you :-)

Yes but you know I have seen many people missing this piece of information and getting into loops. Hence it has become a practise for me to suggest this to many of them which have saved there time ...again as you said this needs the downtime as they need to shut the interface for some time.

Regards

Inayath.

New Member

Re: Added switch to network and err-disabled part of the network

Hi guys, really appreciate all you input, here is the port info for both etherchannels on both stacks:

Stack A

sh int trunk

Port        Mode             Encapsulation  Status        Native vlan

Po1         on               802.1q         trunking      23

Port        Vlans allowed on trunk

Po1         1-4094

Port        Vlans allowed and active in management domain

Po1         1,23,1017-1019

Port        Vlans in spanning tree forwarding state and not pruned

Po1         1,23,1017-1019

sh etherchannel sum

Group  Port-channel  Protocol    Ports

------+-------------+-----------+-----------------------------------------------

1      Po1(SU)          -        Gi1/0/12(P) Gi1/0/24(P)

interface GigabitEthernet1/0/12

description Dot1q Trunk to Nortel Core 1 port 1/45

switchport trunk encapsulation dot1q

switchport trunk native vlan 23

switchport mode trunk

flowcontrol receive on

channel-group 1 mode on

end

interface GigabitEthernet1/0/24

description Dot1q Trunk to Nortel Core 2 port 1/45

switchport trunk encapsulation dot1q

switchport trunk native vlan 23

switchport mode trunk

flowcontrol receive on

channel-group 1 mode on

end

Stack B

sh int trunk

Port        Mode             Encapsulation  Status        Native vlan

Po1         on               802.1q         trunking      23

Port        Vlans allowed on trunk

Po1         1-4094

Port        Vlans allowed and active in management domain

Po1         1,23,1018-1019

Port        Vlans in spanning tree forwarding state and not pruned

Po1         1,23,1018-1019

sh ether sum

Group  Port-channel  Protocol    Ports

------+-------------+-----------+-----------------------------------------------

1      Po1(SU)          -        Gi1/0/12(P) Gi1/0/24(P)

interface GigabitEthernet1/0/12

description Dot1q Trunk to Nortel Core 1 port 2/45

switchport trunk encapsulation dot1q

switchport trunk native vlan 23

switchport mode trunk

flowcontrol receive on

channel-group 1 mode on

end

interface GigabitEthernet1/0/24

description Dot1q Trunk to Nortel Core 2 port 2/45

switchport trunk encapsulation dot1q

switchport trunk native vlan 23

switchport mode trunk

flowcontrol receive on

channel-group 1 mode on

end

Thanks

2652
Views
20
Helpful
19
Replies
CreatePlease to create content