cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
1426
Views
0
Helpful
5
Replies

Rapid pvst issues

askaerr
Level 1
Level 1

Hi,

I'm working for a company that has 2x 6500 chasis switches in the main building as Core switches (CORE1 and CORE2). There are 3 other buildings that house employees (Building 2 and Building 3) and a DR site. The "Core" switches at these other buildings are 3750 switches (stacks of 2). The buildings are connected with 1Gb fibre (MM) leased lines in a square:

loop.jpg

Since a few days we are seeing alot of spanning tree recalculations on the Core switches of Building 2 and 3 which causes alot of network issues for the people in those buildings. More precisely the Gi1/0/1 interface on both core switches of those buildings (see red crosses in picture) are constantly displaying these messages:

Feb  3 10:25:31 Building2-CORE 801113: 690303: Feb  3 10:24:20.544 cet: RSTP(750): Gi1/0/1 rcvd info expired

Feb  3 10:25:31 Building2-CORE 801114: 690304: Feb  3 10:24:20.544 cet: %SPANTREE-2-LOOPGUARD_BLOCK: Loop guard blocking port GigabitEthernet1/0/1 on VLAN0750.

Feb  3 10:25:32 Building2-CORE 801115: 690305: Feb  3 10:24:20.544 cet: RSTP(750): updt roles, information on root port Gi1/0/1 expired

Feb  3 10:25:32 Building2-CORE 801116: 690306: Feb  3 10:24:20.544 cet: RSTP(750): we become the root bridge

Feb  3 10:25:32 Building2-CORE 801117: 690307: Feb  3 10:24:20.552 cet: RSTP(750): updt roles, received superior bpdu on St1

Feb  3 10:25:32 Building2-CORE 801118: 690308: Feb  3 10:24:20.552 cet: RSTP(750): St1 is now root port

Feb  3 10:25:32 Building2-CORE 801119: 690309: Feb  3 10:24:20.552 cet: RSTP(750): synced St1

Feb  3 10:25:32 Building2-CORE 801120: 690310: Feb  3 10:24:20.561 cet: RSTP(750): transmitting an agreement on St1 as a response to a proposal

Feb  3 10:26:21 Building2-CORE 801193: 690383: Feb  3 10:25:10.910 cet: %SPANTREE-2-LOOPGUARD_UNBLOCK: Loop guard unblocking port GigabitEthernet1/0/1 on VLAN0750.

Feb  3 10:26:21 Building2-CORE 801194: 690384: Feb  3 10:25:10.910 cet: RSTP(750): initializing port Gi1/0/1

Feb  3 10:26:21 Building2-CORE 801195: 690385: Feb  3 10:25:10.910 cet: RSTP(750): Gi1/0/1 is now designated

Feb  3 10:26:21 Building2-CORE 801196: 690386: Feb  3 10:25:10.910 cet: RSTP(750): updt roles, received superior bpdu on Gi1/0/1

Feb  3 10:26:21 Building2-CORE 801197: 690387: Feb  3 10:25:10.910 cet: RSTP(750): Gi1/0/1 is now root port

Feb  3 10:26:21 Building2-CORE 801198: 690388: Feb  3 10:25:10.910 cet: RSTP(750): St1 blocked by re-root

Feb  3 10:26:21 Building2-CORE 801199: 690389: Feb  3 10:25:10.910 cet: RSTP(750): St1 is now designated

Feb  3 10:26:21 Building2-CORE 801209: 690399: Feb  3 10:25:10.919 cet: RSTP(750): transmitting a proposal on St1

Feb  3 10:26:21 Building2-CORE 801211: 690401: Feb  3 10:25:10.927 cet: RSTP(750): synced Gi1/0/1

Feb  3 10:26:22 Building2-CORE 801212: 690402: Feb  3 10:25:10.927 cet: RSTP(750): received an agreement on St1

And less than a minute later the same again. This is happening with all VLANs. There's about 125 VLANs and all go over the square.

From what I understand this means BPDU packts are not received in time (2 seconds) and spanning tree starts recalculation. We already asked the provider of the leased lines to test them but they claim nothing is wrong with them. It"s also a bit weird that we are seeing this on 2 different places (physically different locations and lines).


CPU usage looks normal (around 14%) on all switches in this square. Since it's happening on 2 locations I don't think a faulty cable or SFP is causing this.

Any ideas from you guys?

Regards


5 Replies 5

devils_advocate
Level 7
Level 7

Hi

Can you show us the config on the 6500 core switch?

Presumably the links between the buildings are Trunks?

Can post results of the following command on each set of switches:

#show spanning tree vlan 750

Hi,

All links between the buildings are configured as trunks indeed with no VLAN restrictions (all VLANs allowed).

Here is the extract of the command on all 5 switches/stacks:

MAIN-CORE1#sh spanning-tree vlan 750

VLAN0750

  Spanning tree enabled protocol rstp

  Root ID    Priority    8192

             Address     001c.0edc.eaee

             This bridge is the root

             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    8192

             Address     001c.0edc.eaee

             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

             Aging Time 300

Interface           Role Sts Cost      Prio.Nbr Type

------------------- ---- --- --------- -------- --------------------------------

Gi1/3               Desg FWD 4         128.3    P2p

Gi1/4               Desg FWD 4         128.4    P2p

Gi1/5               Desg FWD 4         128.5    P2p

Gi1/6               Desg FWD 4         128.6    P2p

Gi1/7               Desg FWD 4         128.7    P2p

Gi2/22              Desg FWD 4         128.150  P2p

Gi2/23              Desg FWD 4         128.151  P2p

Po10                Desg FWD 3         128.1666 P2p

Interface           Role Sts Cost      Prio.Nbr Type

------------------- ---- --- --------- -------- --------------------------------

Po11                Desg FWD 3         128.1667 P2p

MAIN-CORE2#sh spanning-tree vlan 750

VLAN0750

  Spanning tree enabled protocol rstp

  Root ID    Priority    8192

             Address     001c.0edc.eaee

             Cost        3

             Port        1666 (Port-channel10)

             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    16384

             Address     001c.0edc.daee

             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

             Aging Time 300

Interface           Role Sts Cost      Prio.Nbr Type

------------------- ---- --- --------- -------- --------------------------------

Gi1/3               Desg FWD 4         128.3    P2p

Gi1/4               Desg FWD 4         128.4    P2p

Gi1/5               Desg FWD 4         128.5    P2p

Gi1/6               Desg FWD 4         128.6    P2p

Gi1/9               Desg FWD 4         128.9    P2p

Po10                Root FWD 3         128.1666 P2p

Po21                Desg FWD 4         128.1667 P2p

Building2-CORE1#show spanning-tree vlan 750

VLAN0750

  Spanning tree enabled protocol rstp

  Root ID    Priority    8192

             Address     001c.0edc.eaee

             Cost        7

             Port        1 (GigabitEthernet1/0/1)

             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    33518  (priority 32768 sys-id-ext 750)

             Address     108c.cf03.1d00

             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

             Aging Time 300

Interface        Role Sts Cost      Prio.Nbr Type

---------------- ---- --- --------- -------- --------------------------------

Gi1/0/1          Root FWD 4         128.1    P2p

St1              Desg FWD 100       128.872  P2p

Gi2/0/1          Desg FWD 4         128.55   P2p

Building3-CORE1#show spanning-tree vlan 750

VLAN0750

  Spanning tree enabled protocol rstp

  Root ID    Priority    8192

             Address     001c.0edc.eaee

             Cost        11

             Port        55 (GigabitEthernet2/0/1)

             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    33518  (priority 32768 sys-id-ext 750)

             Address     8cb6.4fb9.7300

             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

             Aging Time 300

Interface        Role Sts Cost      Prio.Nbr Type

---------------- ---- --- --------- -------- --------------------------------

Gi1/0/1          Root BKN*4         128.1    P2p *LOOP_Inc

St1              Root FWD 100       128.872  P2p

Gi2/0/1          Root FWD 4         128.55   P2p

DR-01#show spanning-tree vlan 750

VLAN0750

  Spanning tree enabled protocol rstp

  Root ID    Priority    8192

             Address     001c.0edc.eaee

             Cost        4

             Port        54 (GigabitEthernet2/0/2)

             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    33518  (priority 32768 sys-id-ext 750)

             Address     0013.c37a.e300

             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

             Aging Time 300

Interface        Role Sts Cost      Prio.Nbr Type

---------------- ---- --- --------- -------- --------------------------------

Gi2/0/2          Root FWD 4         128.54   P2p

Gi1/0/1          Desg FWD 4         128.1    P2p

Fa1/0/13         Desg FWD 19        128.15   P2p

Here is the config of MAIN-CORE1 (I removed most interfaces, VLAN interfaces and ACL's from it):

MAIN-CORE1#sh run

Building configuration...

Current configuration : 44402 bytes

!

upgrade fpd auto

version 12.2

no service pad

service timestamps debug datetime msec localtime show-timezone

service timestamps log datetime msec localtime show-timezone

service password-encryption

service sequence-numbers

service counters max age 5

!

hostname MAIN-CORE1

!

boot-start-marker

boot system flash sup-bootdisk:s72033-ipservicesk9-vz.122-33.SXI6.bin

boot system flash sup-bootdisk:s72033-ipservicesk9-vz.122-18.SXF8.bin

boot-end-marker

!

security passwords min-length 1

logging buffered 5000000

no logging console

no logging monitor

!

aaa new-model

!

!

aaa authentication login default group radius local

aaa authentication login CONSOLE local

aaa authentication dot1x default group radius

aaa authorization exec default group radius local

aaa authorization network default group radius local

!

!

!

aaa session-id common

clock timezone cet 1

clock summer-time CEST recurring last Sun Mar 2:00 last Sun Oct 3:00

!

!

no ip domain-lookup

ip tftp source-interface Vlan60

!

ip ftp source-interface Vlan60

ip flow ingress layer2-switched vlan 20

ip sla 3

icmp-echo 172.31.99.5 source-ip X.X.X.X

timeout 2000

frequency 5

ip sla schedule 3 life forever start-time now

ip sla 4

icmp-echo X.X.X.X source-ip X.X.X.X

frequency 5

ip sla schedule 4 life forever start-time now

udld aggressive

udld message time 7

mls qos map cos-dscp 0 10 18 24 34 46 48 56

mls qos

mls netflow interface

no mls acl tcam share-global

mls cef error action freeze

!

!

!

!

!

!

!

errdisable recovery cause udld

errdisable recovery cause security-violation

errdisable recovery cause psecure-violation

errdisable recovery interval 30

diagnostic bootup level minimal

!

spanning-tree mode rapid-pvst

spanning-tree vlan 1,21,166,168,210,842-843 priority 16384

spanning-tree vlan 2-3,7,10,17-18,28,41,44,60,70,78,96,110,112 priority 8192

spanning-tree vlan 121-122,125,127,140,169-170,199,209,213-214 priority 8192

spanning-tree vlan 220-221,253-254,299,318-322,343,350,411,415 priority 8192

spanning-tree vlan 420-421,425,430,450-451,460,500-501,540,602 priority 8192

spanning-tree vlan 650,702,710-716,740,750,895,900-902,910,920 priority 8192

spanning-tree vlan 940 priority 8192

spanning-tree vlan 20 priority 9

spanning-tree vlan 40 priority 8191

!

redundancy

main-cpu

  auto-sync running-config

mode sso

!

vlan internal allocation policy ascending

vlan access-log ratelimit 2000

!

class-map match-any test

class-map match-all DoubleTake_map

  match access-group name DoubleTake

!

!

policy-map DoubleTake_Pol

  class DoubleTake_map

   set ip dscp af41

!

interface Port-channel10

description connection between cores

switchport

switchport trunk encapsulation dot1q

switchport mode trunk

switchport nonegotiate

mls qos trust cos

!

interface GigabitEthernet1/3

description Trunk To access-sw1

switchport

switchport trunk encapsulation dot1q

switchport trunk allowed vlan 17,20,100,112,140,209,300,740,750

switchport mode trunk

switchport nonegotiate

mls qos trust cos

!

interface GigabitEthernet1/4

description Trunk To access-sw2

switchport

switchport trunk encapsulation dot1q

switchport trunk allowed vlan 17,20,27,100,112,209,740,750

switchport mode trunk

switchport nonegotiate

!

interface GigabitEthernet1/5

description Trunk To access-sw3

switchport

switchport trunk encapsulation dot1q

switchport trunk allowed vlan 17,20,70,112,209,221,740,750,901,902

switchport mode trunk

switchport nonegotiate

!

interface GigabitEthernet1/6

description Trunk To access-sw4

switchport

switchport trunk encapsulation dot1q

switchport trunk allowed vlan 10,17,20,28,60,70,100,112,140,209,220,300,343

switchport trunk allowed vlan add 350,540,602,640,641,740,750,840-842,902

switchport mode trunk

switchport nonegotiate

mls qos trust cos

!

interface GigabitEthernet1/7

description Trunk to DR

switchport

switchport trunk encapsulation dot1q

switchport mode trunk

switchport nonegotiate

speed nonegotiate

mls qos trust cos

!

interface GigabitEthernet2/22

description Link to FW1

switchport

switchport trunk encapsulation dot1q

switchport trunk allowed vlan 10,40,165,211-214,220,318,420,451,501,650,651

switchport trunk allowed vlan add 750

switchport mode trunk

logging event link-status

logging event spanning-tree status

load-interval 30

!

interface GigabitEthernet2/23

description link to FW1

switchport

switchport trunk encapsulation dot1q

switchport trunk allowed vlan 78,121,122,124-127,221,319-322,411,415,425,430

switchport trunk allowed vlan add 450,460,461,465,602,712,713,716,750

switchport mode trunk

logging event link-status

logging event spanning-tree status

load-interval 30

mls qos trust dscp

spanning-tree portfast edge

!

interface GigabitEthernet5/1

description Trunk To MAIN-CORE2

switchport

switchport trunk encapsulation dot1q

switchport mode trunk

switchport nonegotiate

mls qos trust cos

channel-group 10 mode on

!

interface GigabitEthernet5/2

description Trunk To MAIN-CORE2

switchport

switchport trunk encapsulation dot1q

switchport mode trunk

switchport nonegotiate

mls qos trust cos

channel-group 10 mode on

!

ip default-gateway X.X.X.X

ip classless

ip forward-protocol nd

ip forward-protocol udp discard

ip route X.X.X.X Y.Y.Y.Y

!

ip http server

ip http access-class 39

ip http authentication local

no ip http secure-server

ip flow-export source Vlan20

ip flow-export version 9

ip flow-export destination X.X.X.X 2000

!

!

ip radius source-interface Vlan20

logging trap debugging

logging source-interface Vlan20

logging X.X.X.X

!

tftp-server sup-bootdisk:s72033-ipservicesk9-vz.122-33.SXH1.bin

snmp-server community X

snmp-server ifindex persist

snmp ifmib ifindex persist

!

radius-server host X.X.X.X. auth-port 1645 acct-port 1646 key 7 Y

radius-server host X.X.X.X auth-port 1645 acct-port 1646 key 7 Y

!

control-plane

!

!

dial-peer cor custom

!

line con 0

exec-timeout 20 0

privilege level 15

password 7 Y

logging synchronous

login authentication CONSOLE

stopbits 1

line vty 0 4

session-timeout 300

access-class vty_mgmt in

transport input telnet

line vty 5 15

session-timeout 60

access-class vty_mgmt in

transport input telnet

!

exception core-file

mac-address-table notification mac-move

ntp clock-period 17179825

ntp source Vlan20

ntp master 1

!

end

What is st1 connected to in buidings 2 & 3 ?

Jon

Hi,

I believe it's the stack port. Buildings 2 and 3 boh have 2x WS-C3750G-12S switches in a stack.

The DR site is also a stack (2x WS-C3750-24TS) but does not show the St1 port in the spanning tree info. I'm not sure why.

Regards


Hello, Esger.

I guess it could be a scaling issue on your 6500.

Please refer to http://www.cisco.com/en/US/docs/solutions/Enterprise/Data_Center/DC_Infra2_5/DCInfra_5.html

Pay attention to "Virtual port per Line card" limitations - it's a valid concern for your design, as you are not pruning VLANs on trunks.

In the configuration provided, you are using about 70 VLANs, so limitation of 1800 virtual ports per line card will be hit with 1800/70=25 ports configured as trunk on the same line card.

PS: to check stability of your fiber, try to use udld with tuned timers (less than STP max_age timer).

PS2: I would build L3 links between buildings, unless you desperately need VLAN extension between buildings.

PS2 added

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: