STP Loop issue on one of the Access switch and spread to Core switch

virgoboy009 · ‎04-17-2010

Hello Friends,

I have faced this routing STP Loop issue on one of the Access switch and spread to other Access switches .

But Impact is increased as the Loop spread over to Core switch.

Could any one please tell me how to avoid this Loop issue by adding neccessary commands.

Apr 15 22:35:19.149 UAE: %RTD-1-ADDR_FLAP: FastEthernet0/24 relearning 35 addrs per min
Apr 15 22:36:19.045 UAE: %RTD-1-ADDR_FLAP: FastEthernet0/24 relearning 22 addrs per min
Apr 15 22:37:19.056 UAE: %RTD-1-ADDR_FLAP: FastEthernet0/24 relearning 25 addrs per min
Apr 15 22:38:19.077 UAE: %RTD-1-ADDR_FLAP: FastEthernet0/23 relearning 15 addrs per min
Apr 15 22:39:19.052 UAE: %RTD-1-ADDR_FLAP: FastEthernet0/15 relearning 22 addrs per min
Apr 15 22:40:19.078 UAE: %RTD-1-ADDR_FLAP: FastEthernet0/16 relearning 6 addrs per min
Apr 15 22:41:19.079 UAE: %RTD-1-ADDR_FLAP: FastEthernet0/19 relearning 14 addrs per min
Apr 15 22:42:19.053 UAE: %RTD-1-ADDR_FLAP: FastEthernet0/19 relearning 14 addrs per min
Apr 15 22:43:19.091 UAE: %RTD-1-ADDR_FLAP: FastEthernet0/4 relearning 8 addrs per min
Apr 15 22:44:19.070 UAE: %RTD-1-ADDR_FLAP: FastEthernet0/4 relearning 8 addrs per min
Apr 15 22:45:19.060 UAE: %RTD-1-ADDR_FLAP: FastEthernet0/15 relearning 9 addrs per min
Apr 15 22:46:19.071 UAE: %RTD-1-ADDR_FLAP: FastEthernet0/24 relearning 89 addrs per min
Apr 15 22:47:19.072 UAE: %RTD-1-ADDR_FLAP: FastEthernet0/24 relearning 51 addrs per min
Apr 15 22:48:19.151 UAE: %RTD-1-ADDR_FLAP: FastEthernet0/24 relearning 34 addrs per min
Apr 15 22:49:19.105 UAE: %RTD-1-ADDR_FLAP: FastEthernet0/9 relearning 16 addrs per min
Apr 15 22:50:19.079 UAE: %RTD-1-ADDR_FLAP: FastEthernet0/23 relearning 5 addrs per min
Apr 15 22:51:19.111 UAE: %RTD-1-ADDR_FLAP: FastEthernet0/24 relearning 8 addrs per min
Apr 15 22:52:19.148 UAE: %RTD-1-ADDR_FLAP: FastEthernet0/24 relearning 40 addrs per min
Apr 15 22:53:19.117 UAE: %RTD-1-ADDR_FLAP: FastEthernet0/24 relearning 40 addrs per min
Apr 15 22:54:19.092 UAE: %RTD-1-ADDR_FLAP: FastEthernet0/9 relearning 15 addrs per min
Apr 15 22:55:19.087 UAE: %RTD-1-ADDR_FLAP: FastEthernet0/9 relearning 6 addrs per min
Apr 15 22:56:19.067 UAE: %RTD-1-ADDR_FLAP: FastEthernet0/16 relearning 6 addrs per min
Apr 15 22:57:19.067 UAE: %RTD-1-ADDR_FLAP: FastEthernet0/16 relearning 7 addrs per min
Apr 15 22:58:19.167 UAE: %RTD-1-ADDR_FLAP: FastEthernet0/16 relearning 6 addrs per min
Apr 15 22:59:19.068 UAE: %RTD-1-ADDR_FLAP: FastEthernet0/16 relearning 9 addrs per min
Apr 15 23:00:19.074 UAE: %RTD-1-ADDR_FLAP: FastEthernet0/15 relearning 15 addrs per min
Apr 15 23:01:19.085 UAE: %RTD-1-ADDR_FLAP: FastEthernet0/15 relearning 7 addrs per min
Apr 15 23:02:19.154 UAE: %RTD-1-ADDR_FLAP: FastEthernet0/24 relearning 53 addrs per min
Apr 15 23:03:19.076 UAE: %RTD-1-ADDR_FLAP: FastEthernet0/24 relearning 46 addrs per min
Apr 15 23:04:19.103 UAE: %RTD-1-ADDR_FLAP: FastEthernet0/9 relearning 16 addrs per min
Apr 15 23:05:19.077 UAE: %RTD-1-ADDR_FLAP: FastEthernet0/9 relearning 19 addrs per min
Apr 15 23:06:19.119 UAE: %RTD-1-ADDR_FLAP: FastEthernet0/19 relearning 5 addrs per min
Apr 15 23:07:10.396 UAE: Can't flush DM 3 queues 06, status F9FD0600 mode FFF9F9FF

Appreciate all your support.

Regards,

KA.

lamav · ‎04-17-2010

Karim:

Without knowing anything about your network's design and history, it's hard to give you advice.

I guess the first thing you should do is ask yourself why this just started happening now? Has anything changed in the topology? Any configuration changes?

Also, are you running spanning tree, if so what flavor? RSTP? MST?

Perhaps you could tell us some more information.

Thanks

Victor

virgoboy009 · ‎04-18-2010

Hello Lamav,

We use PVST+ in our network and other details are as follows-

Core switch : 6500 coreswitch

h2-d1-c2924 flash:c2900xl-c3h2s-mz.120-5.WC9a.bin ( same image on h2-d2 switch)

h2-d2 - c2924 ios ( have observed 5000 address relearning error message for fa 1/1 which is connected fa 2/2 of h2-d1: which is the highest number observed )

After admin down fa 1/1 on h2-d1 port the issue got resolved , we are not sure why Loop occured as there was no changes to the network on that day and neither no switches were added.Due to this all the Vlans on coreswitch used to up and down continuously.

Eigrp is being used as the LAN protocol.

( we have recently added coreswitch 3 weeks back)

( 10.1.1.3 physical IP HSRP Priority:100)

Coreswitch 2

virtual ip-10.1.1.1 |

10.1.1.2 - IP Priority:110) Core switch1 ----- Access Switch (h2-d1)

| | fa 2/2

| |

| | fa 1/1

Access Switch (h2-d2) Core Switch1

| |

Access Switch (h2-d4) --------h2-d5 -------------------- Access Switch (h2-d3)

Core switch log error:-

Apr 15 22:08:51.702 UAE: %STANDBY-3-DUPADDR: Duplicate address 10.1.252.2 on Vlan11, sourced by 0000.0c07.ac0b
Apr 15 22:09:21.762 UAE: %STANDBY-3-DUPADDR: Duplicate address 10.1.253.2 on Vlan15, sourced by 0000.0c07.ac0f
Apr 15 22:09:51.762 UAE: %STANDBY-3-DUPADDR: Duplicate address 10.1.122.2 on Vlan302, sourced by 0000.0c07.ac66
Apr 15 22:10:21.870 UAE: %STANDBY-3-DUPADDR: Duplicate address 10.1.204.2 on Vlan201, sourced by 0000.0c07.acc9
Apr 15 22:10:51.954 UAE: %STANDBY-3-DUPADDR: Duplicate address 10.1.250.2 on Vlan250, sourced by 0000.0c07.acfa
Apr 15 22:11:22.002 UAE: %STANDBY-3-DUPADDR: Duplicate address 10.1.202.252 on Vlan202, sourced by 0000.0c07.acca
Apr 15 22:11:52.022 UAE: %STANDBY-3-DUPADDR: Duplicate address 10.1.249.2 on Vlan249, sourced by 0000.0c07.acf9
Apr 15 22:12:22.330 UAE: %STANDBY-3-DUPADDR: Duplicate address 10.1.250.2 on Vlan250, sourced by 0000.0c07.acfa
Apr 15 22:12:53.250 UAE: %STANDBY-3-DUPADDR: Duplicate address 10.1.2.2 on Vlan7, sourced by 0000.0c07.ac07
Apr 15 22:13:08.374 UAE: %DUAL-5-NBRCHANGE: IP-EIGRP 1: Neighbor 10.1.26.15 (Vlan26) is down: holding time expired
Apr 15 22:13:08.550 UAE: %DUAL-5-NBRCHANGE: IP-EIGRP 1: Neighbor 10.1.250.100 (Vlan250) is down: holding time expired
Apr 15 22:13:08.750 UAE: %DUAL-5-NBRCHANGE: IP-EIGRP 1: Neighbor 10.43.250.2 (GigabitEthernet13/23) is down: holding time expired
Apr 15 22:13:08.782 UAE: %DUAL-5-NBRCHANGE: IP-EIGRP 1: Neighbor 10.1.254.42 (Vlan1) is down: holding time expired
Apr 15 22:13:09.014 UAE: %DUAL-5-NBRCHANGE: IP-EIGRP 1: Neighbor 10.1.251.100 (Vlan251) is down: holding time expired
Apr 15 22:13:09.450 UAE: %DUAL-5-NBRCHANGE: IP-EIGRP 1: Neighbor 10.1.254.20 (Vlan1) is down: holding time expired
Apr 15 22:13:09.614 UAE: %DUAL-5-NBRCHANGE: IP-EIGRP 1: Neighbor 10.5.250.2 (GigabitEthernet13/21) is down: holding time expired
Apr 15 22:13:09.834 UAE: %DUAL-5-NBRCHANGE: IP-EIGRP 1: Neighbor 10.1.254.42 (Vlan1) is up: new adjacency
Apr 15 22:13:09.834 UAE: %DUAL-5-NBRCHANGE: IP-EIGRP 1: Neighbor 10.43.250.2 (GigabitEthernet13/23) is up: new adjacency
Apr 15 22:13:09.834 UAE: %DUAL-5-NBRCHANGE: IP-EIGRP 1: Neighbor 10.1.251.100 (Vlan251) is up: new adjacency
Apr 15 22:13:09.834 UAE: %DUAL-5-NBRCHANGE: IP-EIGRP 1: Neighbor 10.1.254.20 (Vlan1) is up: new adjacency
Apr 15 22:13:09.834 UAE: %DUAL-5-NBRCHANGE: IP-EIGRP 1: Neighbor 10.5.250.2 (GigabitEthernet13/21) is up: new adjacency
Apr 15 22:13:10.250 UAE: %DUAL-5-NBRCHANGE: IP-EIGRP 1: Neighbor 10.1.250.50 (Vlan250) is down: holding time expired
Apr 15 22:13:12.110 UAE: %DUAL-5-NBRCHANGE: IP-EIGRP 1: Neighbor 10.1.26.15 (Vlan26) is up: new adjacency
Apr 15 22:13:12.466 UAE: %DUAL-5-NBRCHANGE: IP-EIGRP 1: Neighbor 10.1.128.3 (Vlan308) is down: peer restarted
Apr 15 22:13:12.694 UAE: %DUAL-5-NBRCHANGE: IP-EIGRP 1: Neighbor 10.1.250.100 (Vlan250) is up: new adjacency
Apr 15 22:13:13.134 UAE: %DUAL-5-NBRCHANGE: IP-EIGRP 1: Neighbor 10.1.251.3 (Vlan251) is down: peer restarted
Apr 15 22:13:13.198 UAE: %DUAL-5-NBRCHANGE: IP-EIGRP 1: Neighbor 10.1.133.3 (Vlan313) is down: peer restarted
Apr 15 22:13:13.254 UAE: %DUAL-5-NBRCHANGE: IP-EIGRP 1: Neighbor 10.1.133.3 (Vlan313) is up: new adjacency
Apr 15 22:13:13.254 UAE: %DUAL-5-NBRCHANGE: IP-EIGRP 1: Neighbor 10.1.251.3 (Vlan251) is up: new adjacency
Apr 15 22:13:13.254 UAE: %DUAL-5-NBRCHANGE: IP-EIGRP 1: Neighbor 10.1.128.3 (Vlan308) is up: new adjacency
Apr 15 22:13:13.318 UAE: %DUAL-5-NBRCHANGE: IP-EIGRP 1: Neighbor 10.1.134.3 (Vlan314) is down: peer restarted
Apr 15 22:13:13.434 UAE: %DUAL-5-NBRCHANGE: IP-EIGRP 1: Neighbor 10.1.134.3 (Vlan314) is up: new adjacency
Apr 15 22:13:14.642 UAE: %DUAL-5-NBRCHANGE: IP-EIGRP 1: Neighbor 10.1.250.50 (Vlan250) is up: new adjacency
Apr 15 22:13:15.034 UAE: %DUAL-5-NBRCHANGE: IP-EIGRP 1: Neighbor 10.1.125.3 (Vlan305) is down: peer restarted
Apr 15 22:13:16.118 UAE: %DUAL-5-NBRCHANGE: IP-EIGRP 1: Neighbor 10.1.126.3 (Vlan306) is down: peer restarted
Apr 15 22:13:16.242 UAE: %DUAL-5-NBRCHANGE: IP-EIGRP 1: Neighbor 10.1.135.3 (Vlan315) is down: peer restarted
Apr 15 22:13:16.298 UAE: %DUAL-5-NBRCHANGE: IP-EIGRP 1: Neighbor 10.1.131.3 (Vlan311) is down: peer restarted
Apr 15 22:13:16.702 UAE: %DUAL-5-NBRCHANGE: IP-EIGRP 1: Neighbor 10.1.125.3 (Vlan305) is up: new adjacency
Apr 15 22:13:17.254 UAE: %DUAL-5-NBRCHANGE: IP-EIGRP 1: Neighbor 10.1.124.3 (Vlan304) is down: peer restarted
Apr 15 22:13:17.994 UAE: %DUAL-5-NBRCHANGE: IP-EIGRP 1: Neighbor 10.1.127.3 (Vlan307) is down: peer restarted
Apr 15 22:13:18.142 UAE: %DUAL-5-NBRCHANGE: IP-EIGRP 1: Neighbor 10.1.254.3 (Vlan1) is down: peer restarted
Apr 15 22:13:19.046 UAE: %DUAL-5-NBRCHANGE: IP-EIGRP 1: Neighbor 10.1.129.3 (Vlan309) is down: peer restarted
Apr 15 22:13:19.454 UAE: %DUAL-5-NBRCHANGE: IP-EIGRP 1: Neighbor 10.1.135.3 (Vlan315) is up: new adjacency
Apr 15 22:13:19.486 UAE: %DUAL-5-NBRCHANGE: IP-EIGRP 1: Neighbor 10.1.124.3 (Vlan304) is up: new adjacency
Apr 15 22:13:19.494 UAE: %DUAL-5-NBRCHANGE: IP-EIGRP 1: Neighbor 10.1.131.3 (Vlan311) is up: new adjacency
Apr 15 22:13:19.610 UAE: %DUAL-5-NBRCHANGE: IP-EIGRP 1: Neighbor 10.1.127.3 (Vlan307) is up: new adjacency
Apr 15 22:13:19.798 UAE: %DUAL-5-NBRCHANGE: IP-EIGRP 1: Neighbor 10.1.254.3 (Vlan1) is up: new adjacency
Apr 15 22:13:19.922 UAE: %DUAL-5-NBRCHANGE: IP-EIGRP 1: Neighbor 10.1.129.3 (Vlan309) is up: new adjacency
Apr 15 22:13:19.946 UAE: %DUAL-5-NBRCHANGE: IP-EIGRP 1: Neighbor 10.1.126.3 (Vlan306) is up: new adjacency
Apr 15 22:13:23.250 UAE: %STANDBY-3-DUPADDR: Duplicate address 10.1.121.2 on Vlan301, sourced by 0000.0c07.ac65
Apr 15 22:13:53.254 UAE: %STANDBY-3-DUPADDR: Duplicate address 10.1.26.2 on Vlan26, sourced by 0000.0c07.ac1a
Apr 15 22:14:04.174 UAE: %DUAL-5-NBRCHANGE: IP-EIGRP 1: Neighbor 10.1.123.3 (Vlan303) is down: retry limit exceeded
Apr 15 22:14:04.182 UAE: %DUAL-5-NBRCHANGE: IP-EIGRP 1: Neighbor 10.1.204.3 (Vlan201) is down: retry limit exceeded
Apr 15 22:14:05.334 UAE: %DUAL-5-NBRCHANGE: IP-EIGRP 1: Neighbor 10.1.204.3 (Vlan201) is up: new adjacency
Apr 15 22:14:07.982 UAE: %DUAL-5-NBRCHANGE: IP-EIGRP 1: Neighbor 10.1.123.3 (Vlan303) is up: new adjacency
Apr 15 22:14:15.110 UAE: %DUAL-5-NBRCHANGE: IP-EIGRP 1: Neighbor 10.1.3.3 (Vlan8) is down: retry limit exceeded
Apr 15 22:14:18.454 UAE: %DUAL-5-NBRCHANGE: IP-EIGRP 1: Neighbor 10.1.3.3 (Vlan8) is up: new adjacency
Apr 15 22:14:23.410 UAE: %STANDBY-3-DUPADDR: Duplicate address 10.1.122.2 on Vlan302, sourced by 0000.0c07.ac66
Apr 15 22:14:24.766 UAE: %DUAL-5-NBRCHANGE: IP-EIGRP 1: Neighbor 10.1.2.3 (Vlan7) is down: retry limit exceeded
Apr 15 22:14:29.022 UAE: %DUAL-5-NBRCHANGE: IP-EIGRP 1: Neighbor 10.1.2.3 (Vlan7) is up: new adjacency
Apr 15 22:14:45.658 UAE: %DUAL-5-NBRCHANGE: IP-EIGRP 1: Neighbor 10.1.19.3 (Vlan19) is down: retry limit exceeded
Apr 15 22:14:46.890 UAE: %DUAL-5-NBRCHANGE: IP-EIGRP 1: Neighbor 10.1.19.3 (Vlan19) is up: new adjacency
Apr 15 22:14:53.490 UAE: %STANDBY-3-DUPADDR: Duplicate address 10.1.4.2 on Vlan12, sourced by 0000.0c07.ac0c
Apr 15 22:15:05.058 UAE: %DUAL-5-NBRCHANGE: IP-EIGRP 1: Neighbor 10.1.19.3 (Vlan19) is down: stuck in INIT state
Apr 15 22:15:05.186 UAE: %DUAL-5-NBRCHANGE: IP-EIGRP 1: Neighbor 10.1.19.3 (Vlan19) is up: new adjacency
Apr 15 22:15:23.490 UAE: %STANDBY-3-DUPADDR: Duplicate address 10.1.19.2 on Vlan19, sourced by 0000.0c07.ac13
Apr 15 22:15:24.902 UAE: %DUAL-5-NBRCHANGE: IP-EIGRP 1: Neighbor 10.1.203.3 (Vlan203) is down: retry limit exceeded
Apr 15 22:15:25.038 UAE: %DUAL-5-NBRCHANGE: IP-EIGRP 1: Neighbor 10.1.18.3 (Vlan18) is down: retry limit exceeded
Apr 15 22:15:26.258 UAE: %DUAL-5-NBRCHANGE: IP-EIGRP 1: Neighbor 10.1.203.3 (Vlan203) is up: new adjacency
Apr 15 22:15:26.626 UAE: %DUAL-5-NBRCHANGE: IP-EIGRP 1: Neighbor 10.1.18.3 (Vlan18) is up: new adjacency

Regards,

KA.

lamav · ‎04-18-2010

Karim:

Once spanning tree converges, and the topology is stable, there should be no loops in the network because the whole purpose of STP is to block any redundant paths that will cause a bridging loop.

That having been said, loops do occur in a L2 network, even with STP enabled, if there are topological changes being made to the network as a result of either human intervention (someone plugging a switch into a data jack at their desk so that they can use more computing devices, for example) or some type of network hardware failure. The most common hardware failure that would cause a bridging loop, which also may not appear in the logs, is a uni-drectional link that results from a bad copper cable, fiber link or switchport itself.

Are you sure that no one is plugging a switch anywhere in the network, most specifically h2-d1? Have you checked the trunk ports for errors? Perhaps you can use different ports on both ends of the access switch's trunk connections. Also, are you running UDLD or UDLD aggressive on your trunk ports?

Why dont you read this document regarding Cisco's best practices for implementing STP? Its for the 6500 switch, but it is largely applicable to any Cisco Catalyst switch because you need to focus on principles of operation and design, not so much on configuration.

http://www.cisco.com/en/US/prod/collateral/switches/ps5718/ps708/prod_white_paper0900aecd805457cc.html

Here is a good link for general configuration.

http://www.cisco.com/en/US/tech/tk389/tk621/technologies_configuration_example09186a008009467c.shtml

HTH

Victor

Leo Laohoo · ‎04-18-2010

In all of your access ports of your swithces, try the following commands:

   spanning-tree portfast
   spanning-tree bpduguard enable
   port security
   switchport port-security aging time 2
   switchport port-security violation restrict
   switchport port-security aging type inactivity
   switchport port-security maximum 1