[SOLVED] Switch keeps flushing Mac addresses causing floods.

[Eran Shirazy]

Hi

We have 3 switches WS-C3560X-24 (version 15.0(2)SE2.) connected in a row, with dot1q Trunk between them, allowing vlan 10 only.

each switch has a 7200 router connected on port 1 - vlan 10 (internet access) running hsrp (only one is active).

and each switch has a FG firewall on port 10 - vlan 10 (three FW are clustered, only one active).

so far so good.

problem started when we inserted an IPS between them as a transparent device,

we took port 5 as IPS-external Access-vlan 10 and port 6 for IPS-internal Access-vlan 20, we also change the FW-wan to vlan 20.

traffic flow is as follows: FW-wan -> (p10) sw-vlan20 (p6) -> internal-IPS-External-> (p5) sw-vlan10 (p1) ->Router.

once we did that we noticed flooding on vlan 10, causing some packet loss toward the internet, which becomes more noticable when traffic increased.

the reason for the flooding is the router's standby (Virtual) and Bia Mac addresses which keep disapearing from the mac table of vlan 10 (and evantually also on vlan20). most important is the virtual mac that serve as DG not showing at all on vlan 20.

extra diags shows that all other mac addresses are steady in the mac table for the same vlan and this happens on all three switches, routers's macs are flushed and re-apearing after few seconds and so on.

the wierdest thing we also discovered is that it happens on all vlans that are trunked to the same router (on different sub-interfaces, of course),

temporary solutions that we found:

Once you shutdown port5 or 6 of the IPS on a specific switch, mac address table on that switch is steady -all is ok.

so instead of shutdown, (cause we want to use the IPS)  we configured "no mac learning vlan 20" and it solved everything.

i know that basically we flooding traffic from FW to IPS-internal and vice versa, but it works.

Nevertheless, we don't understand what has happened in the switch that affected the mac learning proccess, switch logs shows nothing,  Rstp does not show flaps or TCNs, no block port, no incosistent, loopgurad is on and reports nothing.

Sorry for the long previrew, it never happened to us with such topology.

Does anyone have a clue what is going on? is the a bug? a version issue? blame the IPS?

Thanks for any help.

[devils_advocate]

Can you post some configs?

[Eran Shirazy]

Sure

this is the relevant config, that is running now.

removing "no mac address-table learning vlan 20" reproduce the problem

SWITCH-2-ROOT (two other switches are mirrored, except root priority)
=================================================
udld aggressive

spanning-tree mode rapid-pvst
spanning-tree loopguard default
spanning-tree extend system-id
spanning-tree vlan 1-19,21-4094 priority 4096
spanning-tree vlan 20 priority 16384

no mac address-table learning vlan 20

interface GigabitEthernet0/1
description ## ROUTER-INTERNET2 ##
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 10,30,40
switchport mode trunk
load-interval 30
spanning-tree portfast trunk

interface GigabitEthernet0/5
description ## IPS-EXTERNAL ##
switchport access vlan 10
switchport mode access
load-interval 30
no cdp enable
spanning-tree portfast

interface GigabitEthernet0/6
description ## IPS-INTERNAL ##
switchport access vlan 20
switchport mode access
load-interval 30
no cdp enable
spanning-tree portfast
!

interface GigabitEthernet0/10
description ## FW-INTERNET-2 ##
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 20
switchport mode trunk
load-interval 30
spanning-tree portfast trunk

interface GigabitEthernet1/3
description ## TO SWITCH 1 ##
switchport trunk encapsulation dot1q
switchport trunk allowed vlan  10,30,40
switchport mode trunk
load-interval 30
speed nonegotiate

interface GigabitEthernet1/4
description ## TO SWITCH 3 ##
switchport trunk encapsulation dot1q
switchport trunk allowed vlan  10,30,40
switchport mode trunk
load-interval 30
speed nonegotiate

ROUTER-2-ACTIVE  (two other routers are mirrored, except hsrp priority)
==========================================
interface GigabitEthernet0/1
description #### TO SWITCH2 ####
no ip address
load-interval 30
duplex auto
speed auto
media-type rj45
no negotiation auto
!
interface GigabitEthernet0/1.10
description #### COMPANY Network ####
encapsulation dot1Q 10
ip address 1.1.33.242 255.255.255.240
standby 1 ip 1.1.33.254
standby 1 priority 150
standby 1 preempt
!
interface GigabitEthernet0/1.30
description #### VPN NETWORK ####
encapsulation dot1Q 30
ip address 1.1.33.12 255.255.255.240
standby 1 ip 1.1.33.4
standby 1 priority 150
standby 1 preempt
!
interface GigabitEthernet0/1.40
description #### OOB Network ####
encapsulation dot1Q 40
ip address 1.1.33.213 255.255.255.248
standby 2 ip 1.1.33.211
standby 2 priority 150
standby 2 preempt

interface GigabitEthernet0/2

description #### INTERNET ACCESS ####
ip address 2.2.2.2 255.255.255.248

[Eran Shirazy]

Any Idea? Anyone?

[Eran Shirazy]

version 15.0(2) SE2

If it helps anyone, I successfully simulated the problem on other switches in a lab.

After lots of testing, like QinQ and Private Vlan thats had same symptoms, i found that the issue is a version bug, in this version 15.0(2) SE2, which was tested on 5 different switches, the switch had hard time maintening it's "Per Vlan Address Table", mac's seems to leak thoughout the IPS machine (or direct cable), and the switch treats it as normal activity.

prior versions like 12.2(58) or next versions like 15.0(2) SE4, behaves normal, mac addresses are steady, no more flooding, at last.