Network Loop issue

Anand Narayana · ‎07-06-2010

Hi,

I have a peculiar issue since Monday Jul.5,2010. There was network loop which slowly freezing my network & intermittent disconnections to the server sessions from the workstations. When i logged in the core & access switch, i saw the log as

Jul 6 15:11:11 IST: %SW_MATM-4-MACFLAP_NOTIF: Host 0009.0fa3.7168 in vlan 3 is flapping between port Po1 and port Gi0/3
Jul 6 15:11:12 IST: %SW_MATM-4-MACFLAP_NOTIF: Host 0009.0fa3.6fe8 in vlan 3 is flapping between port Gi0/22 and port Po1
Jul 6 15:12:02 IST: %SW_MATM-4-MACFLAP_NOTIF: Host 0009.0fa3.7168 in vlan 3 is flapping between port Po1 and port Gi0/3
Jul 6 15:12:07 IST: %SW_MATM-4-MACFLAP_NOTIF: Host 001f.d0bb.69a4 in vlan 7 is flapping between port Gi0/19 and port Po1

The network setup is a HSRP having 2 core switches & 5 access switches. Each switch is connected one to CORE-1 & the other to CORE-2. CORE-1 & CORE-2 are ether-channeled. This setup is running since June.2009 having no issue. But all of sudden since Jul.5,2010 as mentioned earlier i was getting logs on the switches console. when i ran "sh cdp neighbors" on access switches, it listed me only CORE-1 switches & not CORE-2. But on CORE-1 & CORE-2 when i ran "sh cdp neighbors" it displayed all the access switches. So then digged in to identify the issue. Initially i disconnected all the access switches uplink cables of CORE-1. Network was completely down, even then "sh cdp neighbors" did not display the CORE-2 switch though the interface is UP. Then i connected back the uplink of CORE-1 on all access switches & disconnected CORE-2 uplink on all access switches, there were no loop & network was healthy. Log did not show anything. Then i decided to connect the CORE-2 uplinks on all access to a different port from

Gi 0/20,Gi 0/21,Gi 0/22,Gi 0/23,Gi 0/24 to Gi 0/15,Gi 0/16,Gi 0/17,Gi 0/18,Gi 0/19 no issues where there. Network was healthy no loops & was able to view CORE-1 & CORE-2 when i ran "sh cdp neighbors" on the access switches. Wondering what went wrong on the ports on the CORE-2 switches, when the configuration was same on the other connected ports. Any ideas?

Ganesh Hariharan · ‎07-06-2010

Hi,

I have a peculiar issue since Monday Jul.5,2010. There was network loop which slowly freezing my network & intermittent disconnections to the server sessions from the workstations. When i logged in the core & access switch, i saw the log as

Jul 6 15:11:11 IST: %SW_MATM-4-MACFLAP_NOTIF: Host 0009.0fa3.7168 in vlan 3 is flapping between port Po1 and port Gi0/3
Jul 6 15:11:12 IST: %SW_MATM-4-MACFLAP_NOTIF: Host 0009.0fa3.6fe8 in vlan 3 is flapping between port Gi0/22 and port Po1
Jul 6 15:12:02 IST: %SW_MATM-4-MACFLAP_NOTIF: Host 0009.0fa3.7168 in vlan 3 is flapping between port Po1 and port Gi0/3
Jul 6 15:12:07 IST: %SW_MATM-4-MACFLAP_NOTIF: Host 001f.d0bb.69a4 in vlan 7 is flapping between port Gi0/19 and port Po1

The network setup is a HSRP having 2 core switches & 5 access switches. Each switch is connected one to CORE-1 & the other to CORE-2. CORE-1 & CORE-2 are ether-channeled. This setup is running since June.2009 having no issue. But all of sudden since Jul.5,2010 as mentioned earlier i was getting logs on the switches console. when i ran "sh cdp neighbors" on access switches, it listed me only CORE-1 switches & not CORE-2. But on CORE-1 & CORE-2 when i ran "sh cdp neighbors" it displayed all the access switches. So then digged in to identify the issue. Initially i disconnected all the access switches uplink cables of CORE-1. Network was completely down, even then "sh cdp neighbors" did not display the CORE-2 switch though the interface is UP. Then i connected back the uplink of CORE-1 on all access switches & disconnected CORE-2 uplink on all access switches, there were no loop & network was healthy. Log did not show anything. Then i decided to connect the CORE-2 uplinks on all access to a different port from

Gi 0/20,Gi 0/21,Gi 0/22,Gi 0/23,Gi 0/24 to Gi 0/15,Gi 0/16,Gi 0/17,Gi 0/18,Gi 0/19 no issues where there. Network was healthy no loops & was able to view CORE-1 & CORE-2 when i ran "sh cdp neighbors" on the access switches. Wondering what went wrong on the ports on the CORE-2 switches, when the configuration was same on the other connected ports. Any ideas?

Hi Anand,

In the above message what is connected on those ports if server then check the teaming (load balancing) configuration should be in active/passive from server end and also check any other patch or anything has applied on the server.

and behiavor of this is on same type of OS or server.

Hope to Help !!

Ganesh.H

jorge.calvo · ‎07-07-2010

Hello,

Could you please tell us what switch model is CORE-2 switch? Which module type are you using?

The fact that some ports in the module were not working (no traffic, no CDP) may indicate a physical problem with them or the ASIC.

Regards.

Anand Narayana · ‎07-07-2010

The CORE switches areWS-C3560G-24TS & the access switches are WS-C2960-48TT-L

mahmoodmkl · ‎07-07-2010

The issue seems to be from UDLD,please check the whether both the fiber strands i.e TX&RX are working fine at both the ends.

Thanks

Anand Narayana · ‎07-07-2010

I use Copper cable & not fiber.

mahmoodmkl · ‎07-07-2010

HI,

May be the port ASIC's on core 2 were not working.

I would suggest you to verify your STP topology and confrim whether your STP root bridges are your core switches.

Also verify your etherchannel configs.

Thanks

jorge.calvo · ‎07-07-2010

The WS-C3560G-24TS has 6 ASCIs managing your port this way:

ASIC 2: Ports Gi0/15, Gi0/16

ASIC 5: Ports Gi0/17, Gi0/18, Gi0/19 & Gi0/20

ASIC 6: Ports Gi0/21, Gi0/22, Gi0/23 & Gi0/24

Because all ports between Gi0/15 - Gi0/19 are working and all ports between Gi0/20 - Gi0/24 are not, the ASIC failure could be discarded. It would make sense if Gi0/20 would be working as well.

If you have the possibility of reloading the switch, it would be useful to perform a "show post" after the reload to discard a hardware problem.

Regards.

Anand Narayana · ‎07-07-2010

I think that make sense what Jorge told. Let me try that & shall keep you posted when i get down time for reloading the switch.